NIL BOHRo-SETS, POINCARE RECURRENCE AND 
GENERALIZED POLYNOMIALS 



WEN HUANG, SONG SHAO, AND XIANGDONG YE 

Abstract. The problem which can be viewed as the higher order version of an 
old question concerning Bohr sets is investigated: for any d £ N does the collection 
of {n G Z : S n (S - n) n . . . n (S - dn) ^ 0} with 5 syndetic coincide with that 
of Nild Bohro-sets? 

In this paper it is proved that Nil^ Bohro-sets could be characterized via gen- 
eralized polynomials, and applying this result one side of the problem could be 
answered affirmatively: for any Nil^ Bohro-set A, there exists a syndetic set S 
such that A D {n G Z : S n (5 - n) n . . . n (S - dn) ^ 0}. Note that other side of 
the problem can be deduced from some result by Bergelson-Host-Kra if modulo a 
set with zero density. As applications it is shown that the two collections coincide 
dynamically, i.e. both of them can be used to characterize higher order almost 
automorphic points. 



I. Introduction 

Combinatorial number theory attracts a lot of attention. In such a theory, prob- 
lems concerning Bohr sets are extensively studied, and have a long history which at 
least could be traced back to the work of Veech in 1968 |29j. Bohr sets are funda- 
mentally abelian in nature. Nowadays it has become apparent that a higher order 
non-abelian Fourier analysis plays a role both in combinatorial number theory and 
ergodic theory. Related to this, a higher-order version of Bohr sets, namely Nil^ 
Bohro-sets, was introduced in [IB] . For the recent results obtained by Katznelson, 
Bergelson-Furstenberg- Weiss and Host-Kra see [23j El HE] . 

1.1. Nil-Bohr sets. There are several equivalent definitions for Bohr sets. Here is 
the one easy to understand: a subset A C Z is a Bohr set if there exist m G N, 
a G T m , and an open set U C T m such that {n G Z : net G U\ is contained in A; 
the set A is a Bohr -set if additionally G [/. 

It is not hard to see that if (X, T) is a minimal equicontinuous system, x G X 
and [/ is a neighborhood of x, then N(x,U) =: {n G Z : T n x G Z7} contains 
S — S =: {a — b : a, 6 G 5} with S 1 syndetic, i.e. with a bounded gap. An old 
question concerning Bohr sets is 
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Problem A-I: Let S be a syndetic set of Z, is 5 — S a Bohr -set? 

That is, are the common differences of arithmetic progressions with length 2 
appeared in a syndetic set a Bohr set? Veech showed that it is at least "almost" 
true [29]. That is, given a syndetic set 5CZ, there is some subset N with density 
zero such that (S — S)AN is a Bohr -set. 

A subset A C Z is a M/^ Bohr^-set if there exist a d-step nilsystem (X, T), 
x G X and an open set U C X containing x such that N(x , U) =: {n G Z : 
T™xo G [/} is contained in A. Denote by the familjfH consisting of all Nil^ 
Bohro-sets. We can now formulate a higher order form of Problem A-I. We note 
that {n G Z : S fl (S — n) fl . . . fl (S — dn) ^ 0} can be viewed as the common 
differences of arithmetic progressions with length d + 1 appeared in the subset S. 
In fact, S PI (S — n) fl . . . fl (S — dn) ^ if and only if there is m G S with 
m, m + n, . . . , m + dn G S. 

Problem B-I: [Higher order form of Problem A-I] Let d G N. 

(1) For any Nild Bohr -set A, is it true that there is a syndetic subset S of Z 
with A D {n G Z : S n {S - n) n . . . n (S - dn) ^ 0} ? 

(2) For any syndetic set S, is {n G Z : 5 fl (S* — n) fl . . . fl (S — dn) ^ 0} a Xz/ d 
5o/ir -set? 

1.2. Dynamical version of the higher order Bohr problem. Sometimes com- 
binatorial questions can be translated into dynamical ones by the Furstenberg cor- 
respondence principle, see Section 12.41 Using this principle, it can be shown that 
Problem A-I is equivalent to the following version: 

Problem A-II: For any minimal system (X, T) and any nonempty open set U C X , 
is the set{neZ:Un T- n U ^ 0} a Bohr -set? 

Similarly, Problem B-I has its dynamical version: 

Problem B-II: [Dynamical version of Problem B-I] Let d G N. 

(1) For any Nild Bohr -set A, it is true that there are a minimal system (X, T) 
and a non-empty open subset U of X with 

Ad {nez-.un T- n u n . . . n T- dn u ^ 0}? 

(2) For any minimal system (X, T) and any open non-empty U C X , is it true 
that {neZ:Un T- n U fl . . . fl T- dn U + 0} a Nil d Bohr -set? 

It follows from some result by Bergelson-Host-Kra in [1] that Problem B-II (2) has 
a positive answer if ignoring a set with zero density. In fact, the authors [1] showed: 
Let (X, X, n, T) be an ergodic system and deN, then for all A G X with n{A) > 
the set I = {n G Z : a{A fl T~ n A D . . . fl T~ dn A) > 0} is almost a Nil d Bohr -set, 
i.e. there is some subset N with density zero such that IAN is a Nil^ Bohr -set. 

1 A collection T of subsets of Z (or N) is a family if it is hereditary upward, i.e. F\ C F2 
and Fi e J imply F2 G T . Any nonempty collection A of subsets of Z generates a family 
F{A) :={FCZ:FDAfor some A e A}. 
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1.3. Main results. We will show that Problem B-II(l) has an affirmative answer. 
Namely, we will show 

Theorem A: Let d G N. If A C Z is a Nil d Bohr -set, then there exist a minimal 
d-step nilsystem {X,T) and a nonempty open set U of X with 

Ad {nez-.un T- n U n . . . n T- dn u ^ 0}. 

As we said before for d — 1 Theorem A can be easily proved. To show Theorem A in 
the general case, we need to investigate the properties of J-^o- It is interesting that 
in the process to do this, generalized polynomials (see §H for a definition) appear 
naturally. Generalized polynomials have been studied extensively, see for example 
the nice paper by Bergelson and Leibman [5] and references therein. After finishing 
this paper we even find that it also plays an important role in the recent work by 
Green, Tao and Ziegler [TH]. In fact the special generalized polynomials defined in 
this paper are closely related to the nilcharacters defined there. 

Let J^GPd (resp. J r sGP d ) be the family generated by the sets of forms 

k 

[){n G Z : i^n) (mod Z) G (-«*,<•<)}, 
i=i 

where k G N, Pi, . . . ,P& are generalized polynomials of degree < d (resp. special 
generalized polynomials), and £j > 0. For the precise definitions see §HJ 

The following theorem illustrates the relation between NiLj Bohrn-sets and the 
sets defined above using generalized polynomials. 

Theorem B: Let d G N. Then F dfi = J GPi . 

To prove Theorem B we first figure out a subclass of generalized polynomials 
(called special generalized polynomials) and show that Top d = FsGP d - When d = 1, 
we have J-^o = Fsgpi- This is the result of Katznelson [23], since Fsgp 1 is generated 
by sets of forms Plf =1 {n G Z : naj(mod Z) G (—6$, e»)} with k G N, G K and e« > 0. 

Theorem A follows from Theorem B and the following result: 

Theorem C: Let d G N. If A G FGP d , then there exist a minimal d-step nilsystem 
(X, T) and a nonempty open set U such that 

AD{neZ:Un T-' n U Pi ... Pi T- dn U ^ 0}. 

The proof of Theorem B is divided into two parts, namely 

Theorem B(l): J" d)0 C J = GP d and 

Theorem B(2): F dfi D T GPd . 

The proof of Theorem B(l) is a theoretical argument using nilpotent Lie group 
theory; and the proofs of Theorem B(2) and Theorem C are very complicated con- 
struction and computation where nilpotent matrix Lie group is used. 

Remark 1.1. Our definition of generalized polynomials is slight different from the 
ones defined in [5]. In fact we need to specialize the degree of the generalized 
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polynomials which is not needed in Moreover, our Theorem B can be compared 
with Theorem A of Bergelson and Leibman proved in [5]. 

In [T31 H2] Furstenberg introduced the notion of Poincare recurrence sets and 
Birkhoff recurrence sets. Here is a generalization of the above notion. Let d G N. 
We say that S C Z is a set of d-recurrence if for every measure preserving dynamical 
system (X, X, fj,,T) and for every A G X with fi(A) > 0, there exists n G S 1 such 
that 

/i(AnrMn...nr dn /i) > o. 

We say that S C Z is a set of d-topological recurrence if for every minimal system 
(X, T) and for every nonempty open subset U of X, there exists n G 5* such that 

[/ n T~ n t/ n . . . n r - * 1 */ ^ 0. 

Remark 1.2. The above definitions are slightly different from the ones introduced 
in [IT], namely we do not require n ^ 0. The main reason we define in this way is 
that for each A G T dfl , 06 A Thus {0} U C G J 7 ^ for each C C Z. 

Let J-p j d (resp. ^Bir d ) be the family generated by the collection of all sets of 
d- recurrence (resp. sets of <i-topological recurrence). It is obvious by the above 
definition that Tp i d C Tpir d - Moreover, it is known that for each d G M, Tp Q i d ~=l 
Fpoi d+1 and FBir d ^ ^Bir d+1 [HI- Now we state a problem which is related to 
Problem B-II. 

Problem B-III: Is it true that TBir d = F^q? 

where J 7 ^ is the dual family of J-^o, i-e. the collection of sets intersecting every 
Nild Bohro set. 

An immediate corollary of Theorem A is: 

Corollary D: Let rfeN. Then 

•Fpoi d C J^Bira C J^dfi- 

Note that J-po^ ^ J~Bin [21] • Though we can not prove J-Bir d = 3~dai we wm 
show that the two collections coincide "dynamically" , i.e. both of them can be used 
to characterize higher order almost automorphic points, see §H1 

1.4. Organization of the paper. We organize the paper as follows: In Section 
[21 we give some basic definitions, and particularly we show the equivalence of the 
Problems I, II and III. In Section [3] we recall basic facts related to nilpotent Lie 
groups and nilmanifolds, and study the properties of the metric on nilpotent matrix 
Lie groups. In Section HI we introduce the notions related to generalized polynomials 
and special generalized polynomials, and give the basic properties. In the next three 
sections we show the main results. And in the last section, we state the applications 
of our main results, which will appear in a forthcoming article [22] by the same 
authors. 

1.5. Thanks. We thank Bergelson, Frantzikinakis and Glasner for useful comments. 
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2. Preliminaries 

In this section we introduce some basic notions related to dynamical systems,explain 
how Bergelson-Host-Kra's result is related to Problem B-II and show the equivalence 
of the Problems I, II and III. 

2.1. Measurable and topological dynamics. A (measurable) systemis a quadru- 
ple (X, X , /i, T), where (X, X, /i) is a Lebesgue probability space and T : X — > X is 
an invertible measure preserving transformation. 

A topological dynamical system, referred to more succinctly as just a system, is a 
pair (X, T), where X is a compact metric space and T : X — > X is a homeomor- 
phism. We use p(-, •) to denote the metric on X. 

2.2. Families and filters. Since many statements of the paper are better stated 
using the notion of a family, we now give the definition. See [T| for more details. 

2.2.1. Furstenberg families. We say that a collection J 7 of subsets of Z is a family 
if it is hereditary upward, i.e. F± C F 2 and Fx £ J 7 imply F 2 G J 7 . A family T is 
called proper if it is neither empty nor the entire power set of Z, or, equivalently if 
Z G J 7 and G^ J 7 . Any nonempty collection A of subsets of Z generates a family 
F(A) := {F C Z : F D A for some A G .A}. 

For a family J 7 its dual is the family J 7 * := {F C Z : F n F' ^ for all F' G J 7 }. 
It is not hard to see that J 7 * = {F C Z : Z \ F ^ J 7 }, from which we have that if J 7 
is a family then (J 7 *)* = J 7 . 

2.2.2. Filter and Ramsey property. If a family J 7 is closed under finite intersections 
and is proper, then it is called a filter. 

A family T has the Ramsey property if A = Ai U A 2 G J 7 then Ai G J 7 or A 2 G J 7 . 
It is well known that a proper family has the Ramsey property if and only if its dual 
J 7 * is a filter [13]. 

A subset S of Z is syndetic if it has a bounded gap, i.e. there is X G N such that 
{i,i + 1, ■ ■ ■ , i + N} fl 5 7^ for every « G Z. A subset S 1 is an IP-set, if there is a 
subsequence {p^} of Z such that 

5 D {p h + ...+p in :i 1 < ... <i n ,neN}. 

It is known that the family of all JF*-sets is a filter and each 7F*-set is syndetic 

m. 

The upper Banach density and lower Banach density of S are 

\s n /I IS 1 n 71 

BD*(S) = limsup — : — : — , and BD*(S) = liminf — — — , 

|j|-kx> |7| I-ZI-+00 |7| 

where 7 ranges over intervals of Z, while the upper density of S and the lower density 
of S are 

m/rn ,. \Sn[-n,n]\ . |£n[-n,7i]| 
7)*(S^) = limsup J ^ LL , and = hmmf J L ^ LL . 

n^,oo 2n + 1 n-s>oo 2/Z + 1 

If D*(S) = D*(S), then we say the dens% of 5 is D(S) = D*(S) = D*(S). 
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2.3. A Bergelson-Host-Kra' Theorem and a consequence. In this subsection 
we explain how Bergelson-Host-Kra's result is related to Problem B-II. First we need 
some definitions. 

Definition 2.1. Let k > 1 be an integer and let X = G/Y be a d-step nilmanifold. 
Let be a continuous real (or complex) valued function on X and let a G G and 
b G X. The sequence {(p(a n ■ b)} is called a basic d-step nilsequence. A d-step 
nilsequence is a uniform limit of basic d-step nilsequences. 

For the definition of nilmanifolds see Section [3j 

Definition 2.2. Let {a n : n G Z} be a bounded sequence. We say that a n tends to 
zero in uniform density, and we write UD-Lim a n = 0, if 

M+N-l 

lim sup \ \a n \ = 0. 

N — >+oo MeZ 

n=M 

Equivalently, UD-Lim a n = if and only if for any e > 0, the set {n G Z : \a n \ > e} 
has upper Banach density zero. Now we state their result. 

Theorem 2.3 (Bergelson-Host-Kra). [4J Theorem 1.9] Let (X, X, /i, T) 6e an ergodic 
system, let f G L°°(fi) and let d > 1 fee an integer. The sequence {If(d,n)} is the 
sum of a sequence tending to zero in uniform density and a d-step nilsequence, where 

(2.1) I f (d, n) = J f(x)f(T n x) . . . f(T dn x) d^(x). 

Especially, for any A G X 

(2.2) {I lA (d, n)} = {fi{A n T~ n A n . . . n T- dn A)} = F d + N, 

where Fa is a (i-step nilsequence and iV tending to zero in uniform density. Regard 
Fd as a function Fd : Z — > C. By (20] there is a ci-step nilsystem (Z, S), xo G Z and 
a continuous function G C(Z) such that 

F d (n) = <P(S n x ). 

We claim that <P(xq) > if /~t(A) > 0. Assume that contrary that 4>(xq) < 0. By 
[H] or [61 Theorem 6.15] there is c > such that 

{n G Z : /i(A n T~ n A Pi ... Pi T^^A) > c} 

is an JP*-set. On the other hand there is a small neighborhood V of xo such that 
4>(x) < \c for each x G V by the continuity of 0. It is known that N(x , V) is an 
7P*-set [13] since (Z,S) is distal [2j Ch 4, Theorem 3] or [25]. This contradicts to 
( 12. 2p by the facts that the family of JP*-sets is a filter, each 7P*-set is syndetic and 
N(n) tends to zero in uniform density. That is, we have shown that <P{xq) > if 
fj,(A) > 0. 

Hence if /jl(A) > then for each e > 0, {n G Z : (fi(S n xo) > <P(xq) — |e} is a 
Nil rf -Bohr set. Since {n G Z : |A(n)| > he} has zero upper Banach density we have 
the following corollary 
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Corollary 2.4. Let (X,X,/ji,T) be an ergodic system and d G N. Then for all 
A G X with fJ>(A) > and e > 0, the set 

I = {n G Z : fj,(A D Pi ... Pi T _d M) > 0(x o ) - e} 

is an almost Nil d Bohr -set, i.e. there is some subset M with BD*(M) = such 
that I AM is a Nil d Bohr -set. 

It follows that problem B-II(2) has a positive answer ignoring a set with zero 
density, since for a minimal system (X,T), each invariant measure of (X, T) is fully 
supported. 

2.4. Furstenberg correspondence principle. Let .F(Z) denote the collection of 
finite non-empty subsets of Z. It is well known that 

Theorem 2.5 (Topological case). (1) Let E C Z be a syndetic set. Then there 
exist a minimal system (X, T) and a non-empty open set U C X such that 

{a G J-(Z) : P| T~ n U ^ 0} C {a G .F(Z) : f| (£ - n) ^ 0}. 

(2) For any minimal system (X, T) and any open non-empty set U, there is a 
syndetic set E such that 

{aG^(Z): f)(E-n)^®}C{ae T{%) : f| T~ n U ^ 0}. 

Theorem 2.6 (Measurable case). (1) Let E C Z with BD*(E) > 0. T/ien there 
exists a measurable system (X,X,/j,,T) and A £ X with n(A) = BD*(E) 
such that for all a G .F(Z) 

BD*(f)(E-n))>^f)T- n A). 

(2) Let (X, X, fi, T) be a measurable system and A G X with fi(A) > 0. There is 
a set E with D*(E) > fJ>(A) such that 

(aeJ(Z) : p| (E - n) ^ 0} C {a£7(Z) : /i( p| T~ n [/) > 0}. 

2.5. Equivalence. In this subsection we explain why Problems B-I,II,III are equiv- 
alent. Let T be the family generated by all sets of forms {n G Z : U fl T~ n U fl . . . fl 
rp-dnjj 1 w ith (X, T) a minimal system, [/ a non-empty open subset of X. Then 
it is clear from the definition that 

~F - ~F* 

J Bir^ ' 

Proposition 2.7. For any d G N £/ie following statements are equivalent. 

(1) For any Nild Bohr -set A, there is a syndetic subset S of Z with A D {n G 
Z : 5 n (5 - n) n . . . n (S - dn) 0}. 

(2) For any Nil d Bohr -set A, there are a minimal system (X, T) and a non- 
empty open subset U of X with A D {n G Z : U n T~ n t/ D . . . D T- dn f/ ^ 0}. 

(3) Jflird C J^q . 
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Proof. Let d G N be fixed. (1)=>(2). Let A be a NiLj Bohr -set, then there is a 
syndetic subset S of Z with A D {n G Z : 5 n (5 - n) n . . . n (S - dn) ^ 0}. For such 
S using Theorem I2.5[ we get that there exist a minimal system (X, T) and a non- 
empty open set U C X such that {n G Z : 5 n (5 - n) n . . . n (S - dn) ^ 0} D {n G 

Z:(/nr n (/n...nr dn f/^0}. Thus A d {n g z : unT~ n un. . .(lT~ dn U j£ 0}. 

(2) =>(1) follows similarly by the above argument. (2)=^(3) follows by the definition. 

(3) =>(2). Since -Fb^ C J] and = F*, we have that J 7 * C7] which implies 
that J 7 D F^q. n 

Proposition 2.8. For any deN £/ie following statements are equivalent. 

(1) For any syndetic set S , {n G Z : 5 fl (5 — n) H . . . fl (5 — (in) 7^ 0} is a Nil^ 
BohrQ-set. 

(2) For any minimal system (X,T), and any open non-empty U C X , {n G Z : 
[/ n T~ n £7 n . . . n T~ dn U ^ 0} is a Nil d Bohr -set. 

(3) T B ir a D ^d,o- 

Proof. Let d G N be fixed. (1)=^(2). Let (X, T) be a minimal system and U be a 
non-empty open set of X. By Theorem 12.51 there is a syndetic set S such that 

{n G Z : Sn(£-n)n...n(£-dn) ^ 0} C {n G Z : U fl T~ n U fl . . . fl T~ dn U ^ 0}. 

By (1), {n G Z : 5 n (S - n) n . . . n (5 - dn) ^ 0} is a NiLj Bohr -set, and so is 
{n G Z : [/nr n f/n...nr dn [/ ^ 0}. Similarly, we have (2)^(1). (2)^(3) follows 
by the definition. (3)=>(2). By Corollary D we have = J 7 *, i.e. J" d)0 = J 7 . □ 

2.6. Other related problems. We remark that similar problems can be formu- 
lated replacing syndetic sets by sets with positive Banach density, minimal systems 
by ergodic systems and open non-empty sets by positive measurable sets. 

3. NlLSYSTEMS 

In this section we recall some basic facts concerning nilpotent Lie groups and 
nilmanifolds. Since in the proofs of our main results we need to use the metric in 
the nilpotent matrix Lie group, we state some its basic properties. Note that we 
follow Green and Tao [15] to define such a metric. 

3.1. Nilmanifolds and nilsystems. 

3.1.1. Nilpotent groups. Let G be a group. For g,h G G, we write [g, h] = ghg~ x hT x 
for the commutator of g and h and we write [A, B] for the subgroup spanned by 
{[a, 6] : a G A,b G B}. The commutator subgroups Gj, j > 1, are defined inductively 
by setting G\ = G and Gj + i = [Gj, G]. Let d > 1 be an integer. We say that G is 
d-step nilpotent if Gd+i is the trivial subgroup. 

3.1.2. Nilmanifolds. Let G be a d-step nilpotent Lie group and T a discrete cocom- 
pact subgroup of G, i.e. a uniform subgroup of G. The compact manifold X = G/T 
is called a d-step nilmanifold. The group G acts on X by left translations and we 
write this action as (g, x) i— > gx. The Haar measure \i of X is the unique probability 
measure on X invariant under this action. Let r G G and T be the transformation 
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x i — y tx of X, i.e the nilrotation induced by r G G. Then (X, T, /i) is called a basic 
d-step nilsystem. See [TUl ES] for the details. 

3.1.3. d-step nilsystem and system of order d. We also make use of inverse limits of 
nilsystems and so we recall the definition of an inverse limit of systems (restricting 
ourselves to the case of sequential inverse limits). If (Xj, Ti) ie ^ are systems with 
diam(Xi) < M < oo and fa : Xj + i — > Xi are factor maps, the inverse limit of the 
systems is defined to be the compact subset of Yli^n given by {(xj)jgN : fa(xi+i) = 
Xi,i G N}, which is denoted by lim{Xj}j e N. It is a compact metric space endowed 

with the distance p(x,y) = J2 ieN l/2 l pi(xi,yi). We note that the maps {Tj} induce 
a transformation T on the inverse limit. 

Definition 3.1. [Host-Kra-Maass] [Tj5] A system (X, T) is called a d-step nilsystem, 
if it is an inverse limit of basic rf-step nilsystems. A system (X, T) is called a system 
of order d, if it is a minimal rf-step nilsystem, equivalently it is an inverse limit of 
basic rf-step minimal nilsystems. 

Recall that a subset A C Z is a Nil^ Bohr -set if there exist a rf-step nilsystem 
(X, T), xo G X and an open set U C X containing xq such that X(x , U) is contained 
in A. As each basic rf-step nilsystem is distal, so is a rf-step nilsystem. Hence by 
Definition 13.11 it is not hard to see that a subset A C Z is a Nilj Bohr -set if and 
only if there exist a basic rf-step (minimal) nilsystem (X, T) (or a system (X, T) 
of order rf), x G X and an open set U C X containing x such that N(x ,U) is 
contained in A. Note that here we need the facts that the product of finitely many 
of rf-step nilmanifolds is a rf-step nilmanifold, and the orbit closure of any point in 
a basic rf-step nilsystem is a rf-step nilmanifold [251 Theorem 2.21]. 

3.2. Reduction. Let X = G/r be a nilmanifold. Then there exists a connected, 
simply connected nilpotent Lie group G and rcGa co-compact subgroup such that 
X with the action of G is isomorphic to a submanifold X of X = G/Y representing 
the action of G in G. See [25] for more details. 

Thus a subset A C Z is a iVz^ Bohr^-set if and only if there exist a basic rf-step 
nilsystem (G/Y, T) with G is a connected, simply connected nilpotent Lie group and 
T a co-compact subgroup of G, x G X and an open set U C X containing x such 
that N(xq, U) is contained in A. 

3.3. Nilpotent Lie group and Mal'cev basis. 

3.3.1. We will make use of the Lie algebra g of a rf-step nilpotent Lie group G 
together with the exponential map exp : g — > G. When G is a connected, simply- 
connected rf-step nilpotent Lie group the exponential map is a diffeomorphism [TUl 
[26] . In particular, we have a logarithm map log : G — > g. Let 

exp(X * Y) = exp(X)exp(y), X,Y eg. 
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3.3.2. Campbell-Baker-Hausdorff formula. The following Campbell-Baker-Hausdorff 
formula (CBH formula) will be used frequently 

x (ad X) Pl (ad Y) qi ... (ad X) p "(ad Yf^Y, 

where (ad X)Y = [X,Y]. (If q n = 0, the term in the sum is . . . (ad X) Pn_1 X; of 
course if q n > 1, or if q n — and p n > 1, then the term if zero.) The low order 
nonzero terms are well known, 

x * y =x + y + y] + [x, y]] - -L[y, [x, y]] 
-l[y,[x, [x,y]]]-i[x, [y,[x,y]]] 

+ ( commutators in five or more terms). 

3.3.3. We assume g is the Lie algebra of G over R, and exp : g — >■ G is the 
exponential map. The descending central series of q is defined inductively by 

g W = fl; fl (»+i) = [ flj W] = sp an{[X, Y] : X e g,Y e (n) }. 
Since g is a d-step nilpotent Lie algebra, we have 

= 2 s (2) 2 ... 2 (c ° 2 (d+1) = {0}. 

We note that 

[0 (i) ,0 O) ] Cif + »,Vi,jeN. 

In particular, each g^ fc ^ is an ideal in g. 

3.3.4. Mal'cev Base. 

Definition 3.2. (Mal'cev base) Let G/Y be an m-dimensional nilmanifold (i.e. G 
is a d-step nilpotent Lie group and T is a discrete uniform subgroup of G) and let 
G = G\ D . . . D Gd D Gd+i = {e} be the lower central series filtration. A basis 
X = {Xi, . . . , X m } for the Lie algebra q over M. is called a Mal'cev basis for G/Y if 
the following four conditions are satisfied: 

(1) For each j = 0, . . . , m — 1 the subspace rjj := Span(X J+1 , . . . , X m ) is a Lie 
algebra ideal in g, and hence Hj := exp r)j is a normal Lie subgroup of G. 

(2) For every < i < d we have Gj = Hi i l+ \. Thus = Zq < ^i < • • • < ld-i < 
m — 2. 

(3) Each g <E G can be written uniquely as exp(tiXi)exp(t 2 X 2 ) . . .exp(t m X m ), 
for ti G R. 

(4) T consists precisely of those elements which, when written in the above form, 
have all ti G Z. 

Note that such a basis exists when G is a connected, simply connected (i-step 
nilpotent Lie group [TUl HSJ [26] . 
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3.4. Metrics on nilmanifolds. For a connected, simply connected ci-step nilpotent 
Lie group G, we can use a Mal'cev basis X to put a metric structure on G and on 
G/Y. 

Definition 3.3 (Metrics on G and G/Y). [T5] Let G/Y be a nilmanifold with a 
Mal'cev basis X , where G is a connected, simply connected Lie group and T is a 
discrete uniform subgroup of G. Let <fi : G — > R m with 

g = exp(tiJSfa) . . . exp(t m X m ) h> (t t , ...,t m ). 

We define p = px '■ G x G — > R to be the largest metric such that p(x, y) < 
\4>(xy~ l )\ for all x,y G G, where | • | denotes the £°°-norm on R m . More explicitly, 
we have 

n 

p(x,y) = inf | ^min{|0(x i _ix," 1 )|, \4>(xiX^\)\} : x , . . . , x n G G; x = x, x n = yj. 

i=i 

This descends to a metric on G/Y by setting 

p(xY, yY) := mf{d(x', y') : x', y' G G; x' = x(mod Y);y' — y(mod Y)}. 

It turns out that this is indeed a metric on G/Y, see [T3]. Since p is right-invariant, 
we also have 

p(xY,yY) = Mp(x,m). 

3.5. Base points. The following proposition should be well known. 

Proposition 3.4. Let X = G/Y be a nilmanifold, T be a nilrotation induced by 
a G G. Let x G G and U be an open neighborhood xY in X . Then there are a 
uniform subgroup Y x C G and an open neighborhood V C G/Y x of eY x such that 

N T (xY,U) = N T ,(eY x ,V), 

where T' is a nilrotation induced by a G G in X' = G/Y x . 

Proof. Let Y x = xYx -1 . Then Y x is also a uniform subgroup of G. 

Put V = Ux" 1 , where we view U as the collections of equivalence classes. It is 
easy to see that V C G/Y x is open, which contains eY x . Let n G Nt(xY, U) then 
a n xY G U which implies that a n xYx~ 1 G Ux^ 1 = V, i.e. n G Nt'(gY x , V). The 
other direction follows similarly. □ 

3.6. Nilpotent Matrix Lie Group. 

3.6.1. Let Mrf + i(R) denote the space of all {d + 1) X (d + l)-matrices with real 
entries. For A = (Aj)i<«,j<<i+i e -^d+i(R)> we define 



d+l 

1 A - - 

Hj | 



(3-1) \\A\\ = IA 

Then || ■ || is a norm on Md + i(R) and the norm satisfies the inequalities 

+ < \\A\\ + \\B\\ and \\AB\\ < \\A\\\\B\\ 

for A,BeM d+1 ( 
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3.6.2. Let a = (af)i< k < d ,i<i<d- k +i ^ R d( - d+1 ^ 2 . Then corresponding to a we define 
M(a) with 



M(a) 



a? . 


d- 

. a x 


1 


af 


\ 




d- 


2 






a\ . 


a 2 




a 2 








3 






a\ . 


■ 4~ 




at 2 




. 


■ a d- 


1 






. 


1 








. 







1 


/ 



3.6.3. Let Gd be the (full) upper triangular group 

G d = {M(a) : a- G R, f < fc < d, f < i < d - fc + 1}. 

The group G d is a d-step nilpotent group, and it is clear that for A e G d there exists 
a unique c = (cf )i< k <d,i<i<d-k+i G K d(a!+1)/2 such that A = M(c). Let 

T = {M(h) : h\ G Z, 1 < k < d, 1 < i < d - k + 1}. 

Then T is a uniform subgroup of G^. 

3.6.4. Let a = (af )i< k < d ,i<i<d- k+ i G R d ( d + 1 )/ 2 an d b = (6f )i< fc <d,i<i<d-jb+i G 
M d(d+1)/2 . If c = (cf) 1 < fe < (i)1 < i < (i _ fe+1 G R d ( d+1 )/ 2 such that M(c) = M(a)M(b), 
then 

(3-2) cl = <h~ j V i+k - 3 = a\ + QT «; % k + 6? 

for 1 < k < d and 1 < i < d — k + 1, where we assume a\ = a° = . . . = a° = 1 and 
6? = 6g = ... = 6g = l. 

3.6.5. Now we endow a compatible metric p on G^ and Gd/T. 

Definition 3.5 (Metric on G^). Let / be the (d + 1) x (d + 1) identity matrix. 
Define p : G d x G d — > R such that 

n 

p(A,B) = infl^minlll^xAr 1 -/!!, ||AA-Wll} ■ A , . . . , A n e G; A = A, A n = 

i=i 

Lemma 3.6. For any A, B e G d , 

p(A, B) < \\AB~ 1 - I\\ < \\A - B\\ WB^W and 
\\A-B\ 



\B\ 



< \\AB~ 1 - I\\ < e p^ B )+p^ B f+-+p^ B r _ 1. 



Proof. It is clear that it is sufficient to prove 

(3.3) \\AB- 1 - I\\ < e P(AB)+p(A,B)i+- +P (A,Br _ 1 
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for A, B G Gd- The others are obvious. Let A,Be Gd- For any e > 0, there exist 
A , . . . , A n G Gd, A = A, A n = B such that 

n 

^mmlll^A- 1 - /||, \\A t Al\ - I\\} < p(A,B) + e. 

i=l 

Let U = mmiWAi^A^-lW, \\AiAr\-lW} for i = 1, 2, • • • , n. Note that for C G G d , 
C- 1 = I + Eti( I -C) i - Hence 

d 
i=l 

Moreover, if we set t = min{||C — I\\, 1 1 C 1 — J||}, then 
(3.4) IIC- 1 - J|| < t(l + t + t 2 + ■ • ■ + t^ 1 ). 

Then by flU}, 

||A-iA rl -^ll <t i (i+t i + t* + --. + tt 1 ) 

< t t (l + (p(A, B) +€) + ■■■+ (p(A, B) + e)^ 1 ) 

for % = 1, 2, • • • , n. Thus 

n n 

II A-iA- 1 - J|| < E + B) + e) + --- + (p(A, B) + ey- 1 ) 

i=l i=l 

< ( p (A, 5) + e) + • • • + (p(A, 5) + e) d . 

Now 

1 + \\AB~ 1 - J|| = 1 + \\AoA- 1 - I|| = 1 + H^WA^ 1 - J|| 

= 1 + IKAo^ 1 - Z)^!^ 1 - /) + (A A^ -I) + (A.A- 1 - I)|| 
< 1 + || (^r 1 - /) || || {A x A- 1 - I) || + || (Ao^r 1 - /) || + || (AiA' 1 - 
= (l + \\A A^-I\\)(l + \\A 1 A- l -I\\). 

Continuing this process we get that 

1 + \\AB- 1 - I\\ < (1 + \\A A^ - 7||)(1 + \\A X A? ■■■(!+ IK^A" 1 - 

< e ||A, ) A 1 - 1 -/|| + || J 4i J 4- 1 -/||+--+||A, l _iA- 1 -/|| 
= e (p{A,B)+e)+- + {p(AB)+e) d 

This implies \\AB~ 1 - I\\ < e (p(AB)+e)+-+(p(A,B)+e)" _ L Let e \ o, we get < K^ . 
This ends the proof of the lemma. □ 

Proposition 3.7 (Metrics on Ga and Gd/T). p is a right-invariant metric on Gd- 
This descends to a metric on G d /T by setting 

p(AT,BT) := inf {p(A 7 ,Bi) : 7,7' G T}. 

Since p is right-invariant, we also have 

p(AT, BY) = inf p(A,Bj). 
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Proof. Firstly, it is clear that p : G^ x Gd — > K is a right-invariant, non- negative 
function and for A,B,C G G^, 

p(A, 5) = p(5, A) and p(A C) < p(A, B) + p(£, C). 

By Lemma [3.61 p(A B) = if and only if \\A — B\\ = 0, i.e., A = B. Thus p is a 
right-invariant metric on G^. Moreover by Lemma [3.61 we know that the metric p 
is equivalent to the metric induced by the norm || • || on G^. Thus, p is a compatible 
metric with topology of G^. 

Next we are going to show that this descends to a metric on Gd/Y by setting 

p(AT,BT) := M{p(Ay,B'y') : 7,7' G T}. 

Since p is a right-invariant metric on G^, it is sufficient to show that if p(AY, BY) = 
then AT = BY. Suppose p(^4r, BY) = 0. Since p is right-invariant, inf 7€ r p(^7, B) = 
0. Moreover we can find 7; G Y such that ||5||( e p(^7i,B)+---+p(A 7i ,B) d _-q < | 

for each i G N. By Lemma 13.61 we have 

\\Aji - B\\ < || jB ||( e K^,B)+-+p(A7i,B) d _ !) < 



2*(1+ ll^ll) 
for i G N. Thus for all z < j G N, 

117,-7,11 = \\A- 1 {A( li -B)-A( lj -B))\\ 

< WA^WiWAyt-BW + WA^-BW) 

< " A "^(1 + WA-m) + 2 i (i + p- 1 !!)) < L 

Since 7i, 7? G T, this implies 7, = 7j for z,j G N. Thus 

|U 7l - 511 = \\Aji - B\\ < — \ - — 

11 a 11 11 11 2^(1 + P" 1 !!) 

for any j G N. Hence HA71 — B\\ = 0. So A71 = B and Ar = BY. This ends the 
proof of the proposition. □ 

4. Generalized polynomials 

In this section we introduce the notions and basic properties of (special) general- 
ized polynomials. It will be used in the following sections. 

4.1. Definitions. 

4.1.1. For a real number a G R, let ||a|| = inf{|a — n| : n G Z} and 

\a\ = min{m G Z : |a — m| = ||a||}. 

When studying J-^q we find that the generalized polynomials appear naturally. 
Here is the precise definition. Note that we use f(n) or / to denote the generalized 
polynomials. 
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4.1.2. Generalized polynomials. 

Definition 4.1. Let d G N. We define the generalized polynomials of degree < d 
(denoted by GP d ) by induction. For d = 1, GPi is the collection of functions from 
Z to K containing h a , a G R with /i (n) = an for each n G Z which is closed under 
taking [ ] , multiplying by a constant and the finite sums. 

Assume that GPj is defined for i < d. Then GP^ is the collection of functions 
from Z to R containing GPj with i < d, functions of the forms 

a n po \f 1 (n)]...\f k (n)] 

(with a E R, po > 0, k > 0, fi G GP P; and Y^i=oPi = d), which is closed under 
taking [ ], multiplying by a constant and the finite sums. Let GP= Ug^GPj. 

For example, a\ \a 2 \a3n~\~\ + bin G GPi, and a\ |~a 2 n 2 ] + bi \b 2 R>3 n ll + cin 2 + C2n G 
GP 2 , where a h b h a G R. Note that if / G GP then /(0) = 0. 

4.1.3. Special generalized polynomials. Since generalized polynomials are very com- 
plicated, we will specify a subclass of them, called the special generalized polynomials 
which will be used in our proofs of the main results. To do this, we need some no- 
tions. 

For a G R, we define L(a) = a. For ai,a 2 G R we define L(ai,a 2 ) = ai\L(a 2 )]. 
Inductively, for ai, a 2 , • • • , ae G R (£ > 2) we define 

(4.1) L(a 1 , a 2 , ■ ■ ■ , a e ) = a 1 \L(a 2 , a 3 , • • • , a e )] . 

For example, L(ai,a 2 ,a 3 ) = ai |~a 2 [03] ] . 

We give now the precise definition of special generalized polynomials. 

Definition 4.2. For d G N we define special generalized polynomials of degree < d, 
denoted by SGP rf as follows. SGP d is the collection of generalized polynomials 
of the forms L(n J ' 1 ai, • • • , n-^a^), where 1 < £ < d, 01, ••• , G R,ji, • • • G 
Nwith Etiit< d - 

Thus SGPi = {an : a G R}, SGP 2 = {an 2 ,bn\cn\,en : a,b,c,e G M} and 
SGP 3 =SGP 2 U {an 3 ,an\bn 2 ],an 2 \bn],an\bn\cn}] :a,b,ce R}. 

4.1.4. J-"GP d owd J^SGPa- Let J"GP d be the family generated by the sets of forms 

k 

f){n G Z : Pi(n) (mod Z) G (-€*,€>)}, 
i=i 

where fc G N, Pj G GP d , and q > 0, 1 < % < k. Note that P^n) (mod Z) G (-e i; 
if and only if ||Pj(n)|| < e^. 

Let J^sGPd be the family generated by the sets of forms 

k 

p|{n G Z : P(n) (mod Z) G (-e^)}, 
i=i 

where k G N, Pj G SGPd, and e £ > 0, 1 < i < A;. Note that from the definition both 
JGP d and JsGP d are filters; and FsgPj C Jbp d - 
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4.2. Basic properties of generalized polynomials. 

4.2.1. The following lemmas lead the way to simplify the generalized polynomials. 
Note that for / e GP we let /* = -[/] . 

Lemma 4.3. Let ceR and fx, . . . , f k e GP with fcGN. Then 



cfAi • • • r/fci = n(* - r/ii ) - c (-!) fc E ft -f, 



i=l <!,... ,* fc e{i,*} 

(il,...,ifc)94(*, .-.,•) 



Particularly if k = 2 we ge£ t/iat 

cr/il r/al =c/ir/al -c/t/a + c/ar/il +c(/!- r/il)(/ a - r/ai)- 
Proq/. Expanding n*=i(/i ~ 17*1) we S et that 

II(/i-r/<i)= E ft 1 -ft- 

i=i ii,...,i fc e{i,*} 

So we have 



cr/ii • • • r/*i = fl(/< - r/*i ) - c (-!) fc E /i 1 • • • /i 



i=l H.--.ifc6{l,*} 

(n,...,i fc )^(*,...,*) 



Let c = 1 in Lemma 14.31 we have 
Lemma 4.4. Let fx, fi, . . . , fk e GP. T/ien 



□ 



/i r/ 2 i • • • rAi = (-i) fc_1 n(/< - r/ii) + E /i 1 

i=l >i ifc£{l>*} 

(*l,-.*fc)^(l.*.->*) 

Particularly if k — 2 we have 

Ar/ai = r/ii r/ai + /1/2 - / 2 r/ii - (a - r/iix/a - r/ai)- 

Let = 1 in Lemma [4.31 we have 

Lemma 4.5. Let cGl and / G GP. Then c\f] = cf - c(f -[/!)■ 

4.2.2. In the next subsection we will show that J r GP d = ^SGP d - To do this we use 
induction. To make the proof clearer, first we give some results under the assumption 

(4-2) J~GP d - 1 C J~SGP d - 1 - 

Definition 4.6. Let r e PJ with r > 2. We define 

£ I 

SW r = {[[(wi(n) - \wi(n)]) :£>2,n> l, Wi {n) e GP n and ^n<r} 

i=i i=i 

and 

W r = R - Span{<S>V r }, 
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that is, 

e 

W r = a jPj( n ) : > 1, a,- e R,Pj(n) G <SW r for each j = 1, 2, • • • , £}. 

Lemma 4.7. Under the assumption (14.21) . one /ias /or any p(n) G Wd and e > 0, 
{n G Z : p(n) (mod Z) G (-e, e)} G Fsgp^- 

Proof. Since FsgPj is a filter, it is sufficient to show that for any p(n) = aq(n) and 
\ > 5 > with g(n) G SW^ and a G E, 

{n G Z : p(n) (mod Z) G (-5, 5)} G F S gp^ v 

e 

Note that as g(n) G «SWd, there exist £ > 2, r-j > 1, Wi(n) G GP ri and X] r « — ^ such 

i=l 

that g(n) = n( Uj «( ra ) ~ \ w i( n )~\)- Since £ > 2, one has r\ < d — 1 and so toi(n) G 
i=i 

GP^i. By the assumption flOD, {n G Z : to x (n) (mod Z) G ( — i+o > l+uf)} G 
J-'sGPa-!- By the inequality |g(n)| < |a||wi(n) — [u'i(n)] | for n G Z, we get that 

5 5 

{n G Z : p(n) (mod Z) G (-5, 5)} D {n G Z : IWn) - [Wn)! I G (-- r-r, --)} 

1 + \a\ 1 + |a| 

= {n G Z : Wl (n) (mod Z) G (-t— r-r, TTTl)}- 

1 + \a\ 1 + \a\ 

Thus {n G Z : p(n) (mod Z) G (—5,5)} G J^sGPa-! since {n G Z : u>x(n) (mod Z) G 

Definition 4.8. Let r G N with r > 2. For gi(n), (fe^) G GP r we define 

gi(n) ~ r qr 2 (ra) 
if there exist hi(n) G GP r _i and /i2(w) G W r such that 

92 (rc) = 9i(™) + ^i(^) + /i2(n) (mod Z) 

for all n G Z. 

Lemma 4.9. Let pin) G GP r and g(n) G GP t; r, t G N. T/ien 

(1) p(n)\q(n)] ~ r+t (p(n) - [p(n)])g(n). 

(2) ifqi{n), q 2 (n), ■ ■ ■ , q k (n) G GP t such that q(n) = Y,i=i Qi( n )> then 

k 

p(n)\q(n)] ~ r+i ^ p(n) [g;(n)] . 
»=i 

Proof. (1) follows from Lemma [4.41 and (2) follows from (1). □ 
Definition 4.10. For r G N, we define 

GP; = {p G GP r : {n G Z : p(n) (mod Z) G (-e, e)} G Jsop r for any e > 0}. 
Proposition 4.11. It is clear that 
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(1) Forp(n) G GP r , p(n) G GP' r if and only if -p(n) G GP' r . 

(2) If Pl {n),p 2 {n), ■■■ ,p k (n)e GP' r then 

p(n) = px(n) +p 2 (n) H h G GP r '. 

(3) J r GPd C FsGPa if and only if GP' d = GP d . 

Proof. (1) can be verified directly. (2) follows from the fact that for each e > 0, 
{n G Z : p(n) (mod Z) G (-e,e)} D nf =1 {n G Z : Pi (n) (mod Z) G (-e/k,e/k)}. (3) 
follows from the definition of GP^. □ 

Lemma 4.12. Let p(n),q(n) G GP^ with p(n) ~^ g(n). Under the assumption 
< K2\f . p(n) G GP^ */ and only if q(n) G GP' d . 

Proof. It follows from Lemma [4. 71 and the fact that J-"sGP d is a filter. □ 

4.3. J r G p d = J 7 scp d - 

Theorem 4.13. 7 GPd = JF S GP d - 

Proof. It is easy to see that J r scp d C ^GP d - So it remains to show Tcp d C J r scp d - 
That is, if A G JgPj then there is A' G Fscp d with A D A'. We will use induction 
to show the proposition. 

Assume first d — 1. In this case we let GPi(O) = {g a : a G R}, where g a (n) = an 
for each n G Z. Inductively if GPl(O), . . . , GP x {k) have been defined then / G 
G?Pi(ife + 1) if and only if / G GP X \ {[f j=Q GP 1 {j)) and there are k + 1 \ ] in /. It is 
clear that GP t = U^ =1 GPx(k). life GPi(O) then it is clear that / G GP{. Assume 
that GPi(O), . . . , GP x {k) C GP{ for some k G Z + . 

Let / G GP X (A; +1). We are going to show that / G GP[. If / = f x + / 2 
with /i,/2 G IJi=o ^-fi(0> then by the above assumption and Proposition 14.111 we 
conclude that / G GP{. The remaining case is f — c\f{] + f 2 with c6l \ {0}, 
fx G GP x (k), and / 2 G GPi(O). By Proposition EU and the fact GPi(O) C GP(, 
/ G GP{ if and only if c\f{\ G GP[. So it remains to show c\f{] G GP{. By Lemma 
14.51 we have c\ f{] = cf\ — c(/i — It is clear that c/i G GP x (k) C GP/ since 

/i G GPi(ife) C GP(. For any e > since 

{ne c(A(n) - r/i(n)"|)|| < e} D [n G Z : ||/i(n)|| < 7^}' 

it implies that — c(f\ — \f{\) G GP[. By Proposition 14.111 again we conclude that 
c\f{\ G GP[. Hence / G GP[. Thus GPi C GP[ and we are done for the case d — 1 
by Proposition 14.111 (3). 

Assume that we have proved TGP d _ x C Tsgp^x d > 2, i.e. the assumption ( 14. 2ft 
holds. We define GP d {k) with fc = 0,1,2,.... First / G GP d (0) if and only if 
there is no [ ] in /, i.e. / is the usual polynomial of degree < d. Inductively if 
GPd(O), . . .,GP d (k) have been defined then / G GP fc+ i if and only if / G GP d \ 
(UjU Gp dU)) and th ere are k + 1 \ ] in /. It is clear that GP d = U%L GP d (k). We 
now show GP d (k) C GP' d by induction on k. 
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Let / be a usual polynomial of degree < d. Then f(n) = a n d + fi(n) ~d aon d 
with ft G GP d _ x . By Lemma WM f e GP^ since a n d G SGP d C GP^. This shows 
GPj(O) C GP' d . Now assume that for some k G Z + we have proved 

fc 

(4.3) \jGP d (i)QGP' d . 

Let / G GP d (k + 1). We are going to show that / G GP' d . If / = ft + ft with 
/i 1/2 £ Ui=o^-^(0> then by the assumption fl4.3[) and Proposition 14.111 (2) we 
conclude that / G GP^. The remaining case is that / can be expressed as the sum 
of a function in GPd{0) and a function g G GP d (k + 1) having the form of 

(1) g = c \f 1 ]...\f l ] withc^O, Z>lor 

(2) g = g\{n) \gi{n)~\ . . . \gi{n) \ for any nfZ with #i(n) G SGP r and r < d. 
Since GP d (0) C GP^, / G GP^ if and only if g G GP' d by Proposition EH It 
remains to show that g G GP d . There are two cases. 

Case (1): g = c[ft] . . . [ft] with c ^ 0, I > 1. 

If Z = 1, then g = c[ft] with ft G GP d {k). By Lemma 14.51 we have c|~/i] = 
eft - c(ft - [ft]). It is clear that eft G GP d (k) C GP^ since ft G GP d {k) C GP^. 
For any e > since 

{n G Z : |i - c(ft(rft - [ft(n)])|| < e} 3 {n G Z : ||ft(n)|| < 7^~^}' 

it implies that — c(/i — [ft] ) G GP^. By Proposition 14.111 again we conclude that 
5 = c[ft] G GP^. 

If I > 2, using Lemmas 14.31 and 14.71 we get that 

c[ft] • • • [ft] ^ d -c(-l) 1 J2 fi ■■■ft 1 - 

Since each term of the right side is in GP d (k), g G GP^ by Lemma T4. 121 the assump- 
tion (|13J and Proposition STU (2). 

Case (2): # = <?i(n) [(72 (^)] • • • [5 , «( n )l f° r an y ^ G Z with <?i G SGP r and 1 < r < d 
In this case using Lemmas 14.41 and 14.71 we get that 

i 1 ,...,i l e{i,*} 
(n.--.ii)^(i. *.—.*)>(*<*>—!*) 

Assume ft, . . . , ^ G {1, *} with (ft, . . . , ft) 7^ (1, *,...,*),(*,*,..., *). If there are at 
least two 1 appearing in (ft, ft, • • ■ , ft), then (— l)ftft x . . . g\ l G Ui=o GP&{%). Hence 

{-\y g ?...gYeGP' d 

by the assumption (14. 3p . The remaining situation is that ft = * and there is exact 
one 1 appearing in (ft, . . . , ij). In this case, (— 1) g\ ■ ■ ■ g\ l G GP d {k + 1) is the finite 
sum of the forms din 11 \h\(n) \ . . . [/ij/(n)] with ft > 1 and hx(n) = gi(n); or the 
forms c\h{\ . . . [/i^] or terms in GP' d . 
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If the term has the form din 11 \h\{n)~\ . . . \hi' (n)] with t\ > 1 and foi(n) = gi(n), 

we let <7i (ti) = G^n* 1 |7ti(n)] = a\n tx \g\{n)\ G SGP ri . It is clear cf > r\ > r. If 
n = d, then am* 1 ^(n)] . . . \h Vl {n)\ = g\{n) G GP' d since SGP d c GP^. If n < d, 
then we write 

am^iMl ... r^(n)l =P? ) (n)r^ 1) (n)l... r^(n)l. 

By using Case (1) we conclude that 

(7 ~d finite sum of the forms g^\n)\ g^\n)\ . . . \gf~\rif\ and terms in GP' d . 

Repeating the above process finitely many time (at most rf-times) we get that 
g finite sum of terms in GP' d . Thus g G GP' d by Lemma 14.121 and Proposition 
14.111 (2). The proof is now finished. □ 

5. Proof of Theorem B(l) 

In this section, we will prove Theorem B(l), i.e. we will show that if A G JF^q 
then there are k G N, Pi G GPj (1 < i < k) and e, > such that 

k 

A D G Z : Pi{n) (mod Z) G (-e^e*)}. 

i=i 

We remark that by Section 13. 2} it is sufficient to consider the case when the group 
G is a connected, simply-connected <i-step nilpotent Lie group. 

5.1. Notations. Let X = G/Y with G a connected, simply-connected (i-step nilpo- 
tent Lie group, Y a uniform subgroup. Let T : X — )• X be the nilrotation induced 
by a G G. 

We assume q is the Lie algebra of G over K, and exp : g — > G is the exponential 
map. Consider 

= (1) 2 s {2) 2 ••• 2 (d) 2 (d+1) = {0}. 

We note that 

There is a Mal'cev basis X = {Xi, . . . , X m } for g with 

(1) For each j = 0, . . . , m — 1 the subspace := Span(X J+ i, . . . , X m ) is a Lie 
algebra ideal in q, and hence ifj := exp rjj is a normal Lie subgroup of G. 

(2) For every < i < d we have Gi = Hi i l+ i. 

(3) Each g G G can be written uniquely as exp(t 1 X 1 )exp(t 2 ^2) • • -exp(t m X m ), 
for G E. 

(4) T consists precisely of those elements which, when written in the above form, 
have all ti G Z, 

where G = G\, Gi + i = [Gi, G] with Gd+i = {e}. Thus, there are = Iq < l\ < . . . < 
ld-i < ld = m such that Span — {Xi i+ i, . . . , X m } = g^ +1 ) for i = 0, 1, . . . , d — 1. 

Definition 5.1. Define o(0) = and o(i) = j if + 1 < i < lj, 2 < j < d — 1. 
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5.2. Some lemmas. We need several lemmas. Note that if 

exp^Xi) . . . exp(t m X m ) = exp(«iXi + . . . + u m X m ) 

it is known that [TUl [26] each ti is a polynomial of Ui, . . . ,u m and each it, is a 
polynomial of ti, . . . , t m . For our purpose we need to know the precise degree of the 
polynomials. 

Lemma 5.2. Let {X 1? . . . , X m } be a Mal'cev bases for G/Y. Assume that 
exp(tiXi) . . . exp(t m X m ) = exp{u x X x + . . . + u m X m ). 

Then we have 

(1) Ui = ti for 1 < % < li and if + 1 < i < lj, 2 < j < d then 

u i = ti + ^ ] c k\,...,km,,$l ■ ■ - tm i 

fclo(l) + ... + fc m o(m)<o(i), 
fcQ<m — 2, 0< fe^ , . . . , k m <m 

where ko is the number ofO's appearing in {hi, . . . , k m }. 

(2) ti = Ui for 1 < i < li and if + 1 < i < lj, 2 < j < d then 

U = Ui+ d ku ... :kmJ u k 1 1 ...u 1 ^, 

felo(l)-f...+/c m o(m)<o(i), 
kg •Cm — 2, 0<fc^ ,.. . ,krn <m 

where k is the number ofO's appearing in {k\, . . . , k m }. 

Proof. (1). It is easy to see that if m = 1 then d = 1 and (1) holds. So we may 
assume that m > 2. For s £ {0, 1, ... , m }{ 1 --,™} ; i et {i 

i < . . . < i n } be the collection 

of p's with s(p) 7^ 0. Let 

X s = [X s ( h ), [X s(i2 ), . . . , [X^^X^)]]]. 

For each < p < m let fc p (s) be the number of p's appearing in s (as usual, the 
cardinality of the empty set is defined as 0). Using the CBH formula m — 1 times 
and the condition g( d+1 ) = {0} it is easy to see that (t]Xi) * ... * [t m X m ) is the sum 
of Y1T=\ ti-X-i and the terms 

constant x t qi . . . t qn [X qi , [X q2 , . . . , [X qn _ x , X q J\], m > n > 2, 
i.e. exp(tiXi) . . . exp(t m X m ) can be written as 

m 

exp(^t,X J+ c'/^ s K..tt {s) X s ). 

3 = 1 s e{0,l i ... i m}< 1 .---,™} 
kg (s) <m — 2 

Note that X, C gE^^W^)). Let X s = J2T=i c s,j X J- Thus , <i> • • • , <i = if 
X^^Li kj{ s )°U) > Thus, Mj = ti for 1 < i < l\ and if + 1 < i < lj, 2 < j < d 
then the coefficient of Xj is 

Ui = ti + Ck 1 ,....k m ,iti ■ ■ ■ t^ ■ 

k 1 o(l) + ... + k m o(m)<o(i), 
/cQ<m — 2, 0<^1 > ■ ■ ■ i^m <m 

Note that when fcio(l) + . . . + k m o(m) < o(i) and k^ < m — 2, we have that /c, = 
= . . . = k m — and some other restrictions. For example, when h + 1 < i < h, 
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t* 1 ... = t h t i2 with 1 < i u i 2 < h; and when l 2 + 1 < % < l 3 , t* 1 . . . = t^t^ 
with 1 < ii, i 2 , is < l\ or t^t^ with 1 < ii < li and li + 1 < i 2 < ^2- 

(2) It is easy to see that £, = for 1 < i < Z^. If d — 1 (2) holds, and thus we 
assume that d > 2. We show (2) by induction. We assume that 

t p = u p + ^2 d k' 1 ,...,k' m , P u'l 1 

k' 1 o{l) + ... + k' m o[ m )<o{p), 
k 

for all p with h + 1 < p < i. 
Since 



Ui+i — U+i + Cfc li ... i fc m] j + it^ 1 . . .t^, 

fc 1 o(l) + ... + fc m o(m)<o(i + l), 
fcg<m — 2, 0< fe-j^ , . . . , fc m <m 

we have that 

fc 1 o(l) + ... + fc m o(m)<o(i + l), 
fcQ<m — 2, 0<fc^ , . . . ,fc m <m 

Since o(i + 1) < o(i) + 1 and k < m — 2 we have that if fcio(l) + . . . + k m o(m) < 
o(i + 1) then k p o(p) < o(i) for each 1 < p < m, which implies that k i+ i, . . . , k m = 0. 
By the induction each t p (1 < p < i) is a polynomial of iti, . . . , u m of degree at most 
Y^=i^'j — o(p)(see Equation (I5.ip ) thus 

( -fci,...,fc m ,j+l>l ■ ■ ■ 

k 1 o(l) + ... + k m o(m,)<o(i+l), 
fcQ<m — 2, 0<^,...,fcjn^ m 

is a polynomial of ui, . . . , u m of degree at most Y^=i kp — °(* + !)• Rearranging the 
coefficients we get (2). Note that k^ < m — 2 is satisfied automatically. □ 

Lemma 5.3. Assume that 

x = exp(x i X 1 H h x m X m ) and y = exp(y 1 X 1 ) . . . exp(y m X m ). 

Then 

l\ m 

xy- 1 = expC£2(xi - yi)Xi + ^ i( x i ~ Vi) + p i,i({y P }) + P M X P }^ {v P })) X i), 

i=l i=h + l 

where Pi t i({y p }), P it2 ({x p }, {y p }) are polynomials of degree at most o(i). 
Proof. By Lemma [5.21 we have 

xy~ x = exp(X)exp(F) 

where X = Y^iLi x i X i an d 

m m 

Y = ~ J2 ViXi - J2 ( J2 Ck^...M' m di ■ ■ ■ Vn?) X i- 

i = l i=Zl + l fc^o(l) + ... + fc( n o(m)<o(i), 
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Using the CBH formula we get that 

xy- 1 = exp(X * Y) = exp(X + Y + ±[X, Y] + ±[X, [X, Y]]-\ ) 

m m 

= exp(^(xi -yi 

i=l i=h+l 
h m 

= exp(^(xi - yi)Xi + ^ (i x i ~ Vi) + p iA{Vp}) + p i,2({ x p}i {Vp})) x i) 



i=l i=h+l 



where 



and 



Pi,i({y P }) = - c k[i ... jkLi ^ { ...yfr, 

k' l0 {l) + .. . + k' m o(m)<o(i) , 



PiAM, {y P }) = E 4;;:fA ■ ■ • 4rf 



kQ <m— l,fcg <m — 1 



Note that the reason P,^ has the above form follows from the fact that [gy\ Q^] C 
g(rf+!) = |o} and a discussion similar to the one used in Lemma [5.21 □ 



5.3. Proof of Theorem B(l). Let X = G/T with G a connected, simply-connected 
d-step nilpotent Lie group, V a uniform subgroup. Let T : X — > X be the nilro- 
tation induced by a G G. Assume that A D N(xT, U) with x G X, xT G U and 
U C G/T open. By Proposition 13.41 we may assume that U is an open neighborhood 
of eT in G/T, where e is the unit element of G, i.e. A D iV(er, [/). 

Assume that a = exp(aiXi + • • • + a m X m ), where ai, • • • a m G R. Then 

a n = exp(naiX! + . . . + na m X m ) 

for any n G Z. For ft, = exp(/iiXi) . . . exp(h m X m ) , where h%,- • ■ , /i m G R, write 

a™/*" 1 = exp(piXi + . . . + p m X m ) = exp(w 1 X 1 ) . . . exp(w m X m ). 

Then by Lemma I5TB1 we have pi = na,i — hi for 1 < i < li and if Zj-_x + 1 < i < lj, 2 < 
j < d then 

Pi =na i -/i i + P i ,i({/i p })+ E e fcp , fc /n fcl+ - +fcm ^ .../ij^, 

(5.2) T,^L 1 (kp+k' p )o( P )<o(i), 

0< A: j , . . . ,fc m ,fc^ , . . . , k'jyt <m 
&q <m — 1,/cq <m — 1 

where Pi,i({M) = ~ E c^,...,fc^,<^ • • • ^ m and e fcpifc , = e^^aj 1 . . . 

fc^o(l) + ... + fc( n o(m)<o(i), 
fc' <m-2,0<fc{ ,...,fcf„<m 
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Changing the exponential coordinates to Mal'sev coordinates (Lemma 15. 2[) . we 
get that Wi = ndi — hi for 1 < i < li and if Zj_i + 1 < i < lj, 2 < j < d then 

Wi = Pi+ 4 1) ...,fc m ,^ 1 ■■■p k ™, 

fc^o(l) + ... + fc m o(m)<o(i), 
kQ <m — 2, 0<fc^ >■ ■ ■ j^m <Ti 

in this case using (15. 2 p it is not hard to see that is the sum of —hi and = 
Qi(n, hi, ... , hi_i) such that Qi is the sum of terms 

c(k,k 1 ,...,k l . 1 )n k h k 1 \..h k i r i 1 

with &; + fcio(l) + . . . + ki_io(i — 1) < o(i) (see the argument of Lemma 15.2( 2)). Note 
that if k — then k < m — 2, and if &i = . . . = k m = then k > 1. 

For a given n £ Z, let /ij(n) = [no*] if 1 < i < Zi, and let /ij(n) = \Qi(n, hx(n), . . . , /ij_i(n))] 
if + 1 < i < lj, 2 < j < d. Again a similar argument as in the proof of Lemma 
15.2( 2) shows that hi(n) is well defined and is a generalized polynomial of degree of 
most o(i) < d. For example, if l\ + 1 < i < I2 then 

p i = nai -h i + ^ c{i 1 ,i 2 ,i)h h h i2 + ^ c Ui^) nh h- 

l<ii<?2<'i 1<ji<Zi 

So 

Wi = na- L -hi+ ^2 c(i 1 ,i 2 ,i)h il h i2 + ^ c(ji,i)nh h + ^ d(ii,i 2 ,i)p h p i2 

l<ii<i2<h l<3l<'l l<n<i2<'i 

= nai-hi+ ^2 c(i 1 ,i 2 ,i)h h h i2 + ^ c(ji,i)nh jl 
l<h<i2<h l<ii<ii 

+ 2J d(ii,h,i) (na h - h h ) (na i2 -h i2 ). 

I<i\<i2<h 

Thus if we let hi(n) = \nai\ , 1 < i < l± then if li + 1 < i < Z 2 

/Zi(n) = [naj + 2J c(ii, 2 2 , 2) [na^] [na i2 ] + c C/'i^W na iil 

l<ti<i 2 <ii l<ii<Zi 

+ ^ d(i 1 ,i 2 ,i)(na h - \na h ])(na i2 - \na i2 ])]. 

I<ii<i2<h 

That is, 

/ij(n) = \nai + n 2 a- + 2J c'(ii, i 2) i) [nOjJ [na i2 ] + c 'C?i> O^T^ill 

l<«l<i 2 </i l<il<'l 

is a generalized polynomial of degree at most 2 in n. 

Next we let u>j(n) = naj — /ij(n) = naj — [naj] for 1 < i < l x and if /.,•_. 1 + 1 < i < 
lj, 2<j<d, let 

Wi(n) = Qi{n, hx{n), /ij-i(n)) - /^(n) 

= (5i(n,/ii(n), . . . ,/ii_!(n)) - [^(n, . . . , 
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Since Qi(n, hi(n), . . . , /^-i(n)) is the sum of terms 

c(k,k 1 ,...,k i - 1 )n k h k S(n)...ht- 1 \n) 

with k + fcio(l) + . . . + k m o(m) < o(i) and hi(n) is a generalized polynomial of 
degree of most o(i) < d, we have Wi(n) is a generalized polynomial of degree of most 
o(i) < d. 

Let h(n) = exp(/i 1 (n)X 1 ) • • • exp(h m (n)X m ). Then h(n) G T and a n /i(rz) _1 = 
exp(wi(n)X!) • • • exp(w m (n)X m ). Choose < e << ~ such that 

{ 3 r:^r,er)< e }c(/. 

Then 

A D N(eT, U)D{neZ: p(a n T, eT) < e}. 
We get that (see Definition 13. 3j) 

p(a n T,eT) < p(a n h(n)-\e) < max {|K(n)||}. 

l<fe<m 

So if n G n^LiI 71 ^ : (mod Z) G (— e,e)} then p(aT, er) < e which 

implies that n G iV(er, £/) C A. That is, 

m 

A D G Z : w^n) (mod Z) G (-e, e)}. 

i=l 

This ends the proof of Theorem B(l). 

6. Proof of Theorem B(2) 

In this section, we aim to prove Theorem B(2), i.e. ^ ^GP d - To do this first we 
make some preparations, then derive some results under the inductive assumption, 
and finally give the proof. Note that in the construction the nilpotent matrix Lie 
group is used. 

More precisely, to show ^ ^GP d we need only to prove ^ Fscp d by 
Theorem B. To do this, for a given F G -FsGP d we need to find a <i-step nilsystem 
(X, T), xq G X and a neighborhood U of xq such that F D N(xq,U). In the 
process doing this, we find that it is convenient to consider a finite sum of specially 
generalized polynomials P(n; a%, . . . , a r ) (defined in (16. 4p ) instead of considering a 
single specially generalized polynomial. We can prove that J-^q D J r GP d if and only if 
{n G Z : ||P(n; «i, • • • , atd)\\ < e } ^ -^,o for any an, ■ ■ ■ , G R and e > (Theorem 
16. 7p . We choose (X, T) as the closure of the orbit of T in G^/r (the nilrotation is 
induced by a matrix A G G^), and consider the most right-corner entry zf(n) in 
A n B n with B n G T. We finish the proof by showing that P(n; a.\, ■ ■ • , a^) ~^ (n) 
and {n G Z : ||^f(n)|| < e} G -7-^,0 f° r an Y e > 0. 
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6.1. Some preparations. For a matrix A in G<j we now give a precise formula of 
A n . 

Lemma 6.1. Let x = (x\ )\<k<d,i<i<d~k+i G ]R a! ( d + 1 )/ 2 . For n G N, assume that 
x(n) = (^(n))i<^i<iW^ 1)/2 safe^es M(x(n)) = M(x)», ffren 

(6.1) x*(n) = ©P^x; », fc) + (™)P 2 (x; i, fc) + • • ■ + QP fe (x; i, fc) 

/or 1 < < d and I < i < d — k + 1, where (?) = w ( w ~ 1 )"J"~ fc + 1 ) y or n,k £N and 
P?( x ;2,/c) = a;j 1 a;i+ Sl 2;j+ Sl+S2 • • • a; i + Sl+S2+ ... +S£ _ 1 

(«1.«2.- ,^)S{1,2,-.- ,k}* 
s l+ s 2^ hs^ — fc 

/or 1 < jfe < d, 1 < i < d - k + 1 and 1 < £ < k. 

Proof. Let xf — 1 and x°(m) = 1 for 1 < z < and m G N. By ( 13. 2p . it is not hard 
to see that 

k 

(6.2) x\[m + 1) = z?~V) ■ x j i+k _. 

j=0 

for 1 < k < d, l<i<d-k+l and m G N. 

Now we do induction for k. When k = 1, x}(l) = x\ and x\{m + 1) = x\{m) + x\ 
for m G N by (16. 2p . Hence 2?j(n) = raj = (™)Pi(x; i, 1). That is, (16.11) holds for 
each 1 < i < d and n G N if k — 1. 

Assume that 1 < £ < d - 1, and (15TTD holds for each 1 < k < £, 1 < i < d - k + 1 
and n G N. For k = I + 1, we make induction on n. When n = 1 it is clear 

x\ (1) = a;* = g)P 1 (x; i, k) + Q)P 2 (x; z, *) + • • • + g)P fc (x; z, fc) 

for 1 < i < d- k + 1. That is, (JOJ holds for jfe = 1 + 1, 1 < i < d- & + 1 and n = 1. 
Assume for n = m > 1, (16. ip holds for k = i + 1, 1 < i < d — k + 1 and n = m. For 
n = m + 1, by JO} 

fc-i 

aftn) = x^m) + ( ^ x k r j (m) ■ x J i+k _^j + x\ 

3=1 

fc— 1 k—j 



*<V) + ( ^(E (™)P r (x; i, fc - j)) ■ 4^)) + x\ 

3=1 r=1 
k—1 k—r 

^H+(E(E p ^ x ;^-i)4-i)(;))+^ 



r=l j'=l 
fc-1 fc-1 



r=l j=r 

for 1< i < d - k + 1. Note that 
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fc-1 fc-1 

j=r j=r (si,--- ,*r)e{l,2,». ,k-l} r 

s x H \-s r =j 



Si SO S r k—j 

X j+Sl X i + Sl + --- + S r _l X i+J 



which is equal to 

Jj i+s 1 "S+siH hs r _i i+siH hs r _i+s r J r+1 L i 'V 



(si.- .ar>«r+l)e{l,a,- ,fc-l}H-l 
S 1 + S 2H hsr+s r + l— k 



for 1 < r < k — 1 and 1 < i < d — k + 1. Collecting terms we have 



fc-i 

x1(n) = sf(ro) + (£p r+1 (x; *,*)(?)) +x* 

r=l 
fc 

= xf(m) + ( J> r (x;z,fc)( r ™)) +Pi(x;i,fc) 

r=2 

m fc 

= (£>(x; *,*)(?)) + (£)P r (x;i,fc)( r ™)) +Pi(x;i,fc). 

r=l r=2 

Rearranging the order we get 

fc 

x*(n) = (m + l)P!(x; z, fc) + ^ ((-) + ( p ™))P r (x; i, fc) 

r=2 

k k 

= ^( m r +1 )P(x;z,fc) = ^C)P r (x;z,fc) 

r=l r=l 

for 1 < z < cf — fc + 1. This ends the proof of the lemma. □ 
Remark 6.2. By the above lemma, we have 

P x (x; i, k) = x\ and P fc (x; z, fc) = x\x\ +l ■ ■ ■ 4+fc-i 
for 1 < fc < d and l<z<d-fc + l. 

6.2. Consequences under the inductive assumption. We will use induction to 
show Theorem B(2). To make the proof clearer, we derive some results under the 
following inductive assumption. 

(6-3) Fd-i,Q ^ Fgp^i 

where d G N with d > 2. For that purpose, we need more notions and lemmas. 
The proof of Lemma 16.31 is similar to the one of Lemma 14.71 where Wd is defined in 
Definition 14.61 

Lemma 6.3. Under the assumption ( 16. 31) . one has for any p(n) G Wd and e > 0, 
{jiGZ: p(ri) (mod Z) G (— e, e)} G Fd-i,Q- 
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Definition 6.4. For r G N, we define 

GP r = {p(n) G GP r : {n G Z : p(n) (mod Z) G (-e, e)} G J> j0 for any e > 0}. 

Remark 6.5. It is clear that for p(n) G GP r , p(n) G CP,, if and only if —p(n) G GP r . 
Since J-" r)0 is a filter, if Pi(n),p 2 (n), ■ ■ ■ ,Pk(n) G GP r then 

+ P2(n) H hpfc(n) G GP r . 

Moreover by the definition of GP d , we know that J 7 ^ D Fcp d if an d only if GP d = 
GP d . 

Lemma 6.6. Let p(n) , q(n) G GPd with p(n) ~ d q(n). Under the assumption (16. 3j) . 
p(n) G GPd i/ and cm% if q{n) G GPd. 

Proof. This follows from Lemma 16.31 and the fact that J-d t o is a filter. □ 
For «i, 0:2, . . . , a r G IR, r G N, we define 
P(n; aii, ol 2 , • ■ • , a r ) 

(6.4) =s E (-d'-'l ^n^.^n^.-.^n^j 

^=1 jL-.j/SN Ji ' n=l r 2 =l J£ ' r f =l ^ X J l 

JlH M^=r 

where the definition of L is given in (14. ip . 

Theorem 6.7. Under the assumption (16.31) . the following properties are equivalent: 

(1) FdflO) J 7 ap d . 

(2) P(n; ai, 0:2, • • • , € GPd /or any ai, a 2 , • • • , «d G M ; iaai is 

{n G Z : P(n; a 1; a 2 , • • • , a d ) (mod Z) G (-e, e)} G J^o 

/or any ai, 02, ■ - • , ay G M and e > 0. 

(3) SGPd C GP d . 

Proof. (1) =>- (2). Assume J-^o 3 ^GP d - By the definition of GPd, we know that 
J"d,o D J~GP d if and only if GPd = GPd. Particularly P(n; ax, a 2 , ■ ■ • , «d) € GPd for 
any ax, a 2 , • • • , «d £ 

(3) =>- (1). Assume that SGPd C GP d . Then J^o ^ J r SGP d - Moveover J^o 3 
Fscp d = Fcp d by Theorem |4.13[ 

(2) =>■ (3). Assume that P(n; ax, a 2 , • • • , «d) £ GPd for any a 1; a 2 , ■■■ ,aj G R. 
We define 

^d = {(ji, J2, • • ■ Ji) ■ I e {1,2, • ■ ■ ,d}, ji, j 2 , • ■ ■ , j£ G N and = d}. 

t=i 

For (ji, j 2 , ■ • • Jt), (n,r 2 , ■ ■ ■ ,r s ) G £ rf , we say (ji,j 2 ,--- Jt) > (n,r 2 ,--- ,r s ) if 
there exists 1 < t < £ such that j t > r s and ji = r^ for i < t. Clearly (Ed, >) 
is a totally ordered set with the maximal element (d) and the minimal element 
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For j = (ji, j 2 , ■ • • , je) e S d , put 

£(j) = {LCn^ai, • • ■ , n^a £ ) : a x , • • • , a/ G R}. 

Now, we have 

Claim: £(s) C GP^ for each s G 'Ed- 
Proof. We do induction for s under the order >. First, consider the case when 
s = (d). Given a x G R, we take «i = 1, a 2 = 2, • • • a d -i = d — 1 and = da%. Then 

for any 1 < jx < d — 1, ^ F{ ct t G Z for jiGZ. Thus 

P(n; oti, ot 2 , ■ • • , a d ) = L(— jj a t ) = L(n d ai) (mod Z) 

for any neZ. Hence L{n d ai) G GP^ since P(n; «i, a 2 , ■ ■ ■ , a^) G GP^. Since a x is 
arbitrary, we conclude that C((d)) C GPd- 

Assume that for any s > i = • • • ,4) G we have £(s) C GPd- Now 
consider the case when s = i = (ii, • • • There are two cases. 

The first case is k = d, i± = i 2 = ■■■ = id = 1- Given ai, a 2 , ■ ■ ■ , o,d G R, by 
the assumption we have that for any (j u j 2 , ■■■ ,je) > i, £((ji, j 2 , • • • ,je)) C GP d . 
Thus 

e e (- i )^ iL (^n a ^'^n a ^----'^n a ESi t+) -j 

£=1 Ji,-,^eN Ji ' n=l J " ! ' r 2 =l J r t =\ 

21 H hif =r 

belongs to GP^ by the Remark 16.51 This implies 

P(n; ai, a 2 , • • • , ad) — L(nai,na 2 , • ■ ■ , nad) G GP^ 
by (16. 4p . Combining this with P(n; ai, a 2 , • • • , a<f) G GP^, we have 

L(nai,na 2 , • • ■ , na^) G GP d 

by Remark [6.51 Since ai, a 2 , ■ ■ ■ , a d G R are arbitrary, we get £(i) C GP^. 

The second case is i > (1, 1, • • ■ ,1). Given a%, a 2 , • ■ ■ , a& G R, for r = 1, 2, • • • , k, 
we put a^pi jf+(l = h for 1 < fa < i r — 1 and a j2l~li t +i r = ^ rCLr - 

By the assumption, for (ji,j 2 , ■ ■ ■ ,je) > i, 

L (,-T 11 ^' 77 11 77 11 "EfclA+rJ G GPd ' 

J1 ' ri = l Ji ' r 2 =l J n=l 

For (ji,j 2 , ■ ■ ■ ,je) < i, there exists 1 < u < k such that j t = i t for 1 < t < u — 1 
and i u > j u . Then 
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When u = 1, by fl63|) . 

\TT 11 ari ' 7J 11 a ££i*+rJ G Z 
for any fiGZ. Hence 

/fl^ 1 fylt -j—j- \ 

HTT 11 «n. "I 11 a E£i*+rJ G GPd - 

When u > 1, write /3„ = n^=i ^E" - / h+rv f° r ^ = 1? 2, ■ ■ • , Then (3 U = 1 and 

[L(n^/3 u ,^ +1 /3 n+1 ■ ■ • , n*ft)l = L(n 3 '"/3 U , n^(3 u+1 ■ ■ • , ra*ft). 
Moreover, 



v 1 - — ^ n.-« 



/n^ -,-r TV* -,-r 71-" -pr 

J1 ' ri=l Ju ' r„=l J£ ' r*=l 

is equal to 

L (n^ft, • • • ,n j ~p u , • • • , ni'Pt) = L (n jl fix, ■ ■ ■ , /3 M _i[ L(n J « /3 U , • • • , n*ft)"|) 
which is equal to 

L (n^ft, ■ ■ ■ , n^- 1 Pu-iLin^ /3 U , n ju+1 (3 u+1 • • ■ , n j 'Pt)) 
= L (n^, ■ • • ,n^- + ^/3 u _ 1 /3 u [L(^+ 1 /3 u+1 • ■ ■ , n*ft)"|) 
= L (n jl /3 h • • • , n^- 1+ ^/3 u _!/3 u , n^ +1 /3 u+1 • • • , n*ft) G GP d 

since (ji, ■ • • , j u _ 2 , j u _i + j u ,j u +i, •■■ ,jt)> i- 

Summing up for any j = (ji, • ■ ■ ,je) £ Srf with j 7^ i, we have 

n ^> • • • > 7T n a Er=i i i«+ri' ' 77 n e 

n=l Ju - r„=l r 4 =l 

Combining this with P(n; «i, • • • , ct^) G GP^, we have 
Lyn n ai, n t2 a 2 , ■ ■ • , n^a^ 

= L {h n a ^ n ^ 1+ r 2 , , |j n « E ^ it+r j e cp d 



ri=l 2 r 2 =l B r fc =l 



by (16. 4p and Remark (16. 5p . Since ai, • • • , G R are arbitrary, C GP^. □ 
Finally, since SGP^ = Ujgs d £(J)> we have SGP^ C GPd by the above Claim. □ 
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6.3. Proof of Theorem B(2). We are now ready to give the proof of the Theorem 
B(2). As we said before, we will use induction to show Theorem B(2). Firstly, for 
d = 1, since J-'gpi — Psgpi and J-^o is a filter, it is sufficient to show for any a G K 
and e > 0, 

{n G Z : an (mod Z) G (— e, e)} G Pi,o- 
This is obvious since the rotation on the unit circle is a 1-step nilsystem. 

Now we assume that Pd-i,o ~~> J~GP d _i_i i- e - the the assumption ( 16. 3 p holds. By 
Theorem 16 .7\ to show Pd j0 D Pgp^, it remains to prove that P(n; a\, a^, ■ • ■ , ay) £ 
GPrf for any ai, 0:2, ...,^61, that is 

{n G Z : P(n; a x , a 2 , • • ■ , a d ) (mod Z) G (-e, e)} G P di0 

for any ai, a 2 , 
Let «i, a 2 , • 



«i for i = 1,2, 



, ay G R and e > 0. 

a d G E and choose x = {x%) 1 < k < d , 1 < i < d - k+1 G 
, d and 



Dd(d+i)/2 with 1 



for 2 < Jfe < d and 1< i < d - Jfe + 1. Then 



/ 1 ai 

1 a 2 



A = M(x) 



\ 








V 

For n G N, if x(n) = (a;* (n))^^^^^ G M^ +1 )/ 2 satisfies M(x(n)) = A n , 
then by Lemma 16.11 and Remark \6.2\ 



otd-i 
1 «d 
1 / 



(6.6) 



a£(n) 



©P,(x;U) = © 



,1 

i+fc-1 







a^ai+i • • • a i+fc _i 



for 1 < k < d and l<i<d-k + l. 

Now we define f}{n) = \x}(n)~\ = \nai\ for 1 < % < d and inductively for 
k = 2, 3, • • • , d define 



(6.7) 



ff(n) 



fc-i 

3=1 



i+k—j 



n) 



for 1 < i < cf — /c + 1. Then we define 

4( n ) = x K n ) -fi( n ) 
for 1 < % < d and inductively for k = 2, 3, • • • , d define 

fc-i 

(6-8) z*(n) = xf(n) - (j>* '(n)/^ ») - /*(n) 

j'=i 

for 1 < i < d - k + 1. 

It is clear that zf(n) G GPfc for 1 < k < d and 1 < i < d — k + 1. First, we have 

Claim: P(n; «i, a 2 , ■ ■ • , a d ) ~ d zf (n). 

Since the proof of the Claim is long, the readers find the proof in the following 
subsection. Now we are going to show zf(n) G GP d . 
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Let X = Gd/r be endowed with the metric p in Lemma 13.71 and T be the nil- 
rotation induced by A G G^, i.e. i?r i-)- ABY for £> G G^. Since G^ is a d-step 
nilpotent Lie group and T is a uniform subgroup of G d , (X, T) is a <i-step nilsystem. 
Let xq = T G X and Z be the closure of the orbit orb(xo, T) of e in X. Then (Z, T) 
is a minimal <i-step nilsystem. We consider p as a metric on Z. 

For a given 77 > choose 6 > such that e' 5+ ' 52+ "' +5 ' i — 1 < mm{|,77}. Put 

U = {z E Z : p(z, x ) < 5} 

and 

S = {n G N : p(A n r, r) < 5} = {n G Z : T n x G [/}. 

Then S G since (Z,T) is a minimal (i-step nilsystem. In the following we are 
going to show that 

{m G Z : zf(rn) (mod Z) G (-77,77)} D 5. 
This clearly implies that {m G Z : zf{m) (mod Z) G (—77,77)} G J-^o since S G J-^o- 
As 77 > is arbitrary, we conclude that zf (n) G GPj- 

Given n G S, one has p(A n T, T) < 5. Since p is right- invariant and T is a group, 
there exists B~ 1 G r such that p(A n , B~ l ) < 5. Take h(n) = (— h\ {ri))\<k<d ,i<i<d-k+i G 
Z d(d+i)/2 with M(h(n)) = 5„. By fl3T3|l . 

(6.9) ||A n 5 n -/|| <e 5+52+ - +5d -l<min|ir7}. 

Let y(n) = (yf(n)) 1 < k < d , 1 < i < d ^ k+1 G M d ( rf+1 )/ 2 such that 

M(y(n)) = A n £ n = M(x(n))M(h(n)). 

By 

(6.10) y*(n) = aftn) - (^^W^W™)) - tf(n) 

i=i 

for 1 < A; < d and l<i<d — + Thus 

(6.11) <min{-, 77} 

for 1 < k < d and 1 < i < d — /c + lby (16. 9p . Hence ^K^) = f 2 -^ C 72 )! = l~ nQ! il f° r 
1 < i < d and 



(6.12) h1(n) 



k-l 



■r'; (n) - a;, fc j (n)h j i+l ._ .( // 1 



for 2 < A; < d and 1 < i < d - k + 1. 

Since hj(n) = \nai\ = f}{n) for 1 < i < d, one has /if(n) = /*(n) for 2 < A; < d 
and 1 < i < d - fc + 1 by (Q and fl6TT2|) . Moreover by ([618]) and flBTTOjl . we know 

(n) = y\ (n) for 2 < k < d and 1 < i < g? — k + 1. Combining this with (16. lip . 
|^ fc (n)| < min{|,r7} for 1 < k < d and 1 < i < d — k+1. Particularly, |^f(77)| < r\. 
Thus 

n G {m G Z : zf(m) (mod Z) G ( — 77, 77)}, 
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which implies that {m G Z : zf(m) (mod Z) G (—T],r])} D S. That is, #x(n) G GPrf. 

Finally using the Claim and the fact that zf (n) G G-Pd we have P(n; «i, a 2 , • • • , atd) £ 
GPd by Lemma 1631 This ends the proof, i.e. we have proved F^o 3 Tgp a - 

6.4. Proof of the Claim. Let 

fc-i 

uj(n) = z*(n) + /*(n) = xf (n) - ^^H^-iW 

for 1 < k < d and 1 < i < d — k + 1. Then 

tf(n) = M(»01 
for 1 < k < d and 1 < i < d - k + 1. 

We define U(n;ji) = jj- Yll=i a r for 1 < ji < d. Then inductively for £ = 
2, 3, • • ■ , d we define 

U(n;j 1 ,j 2 ,--- ,jt) = {U(n;ji,--- - [C/"(n; j x , ■ ■ ■ Jt-i)])— IJ^Efciit+r 

= (U(n;j 1} --- ,j t _ x ) - \U(n;ji,--- ,^_ 1 )])L(— jj a^Ja+r/) 

for ji,j 2 , ■ • • ,je > 1 and ji + • • • + < d (see fl4.ll) for the definition of L). 
Next, U(n;d) = ^fla r = L(^f[c 

r=l r=l 

ji + j2-\ h j£ = rf, by Lemma Ej^l) 

U(n; j x , j 2 , ■ ■ ■ ,jt) = (U(n;j 1 ,--- ,j £ -i) - \U(n;j ir -- ,^_i)])L(^-y JJ a E *-i jt+J7 ) 



d d d d 

Next, E/(n; d) = | [] «r = J] a r ) and for 2 < £ < d, j 1 ,j 2 , ■ • • G N with 

r=l r=l 



n 3t 



IT*** 

-d (7(n;ji,-,jM)[%n«Et1^)l 

r £ =l 

which is 

= (U(n;ji, ■ ■ ■ ,j e - 2 ) - \U(n;j u ■ ■ ■ ,j e _ 2 )])x 

n k-i 3 ±± n k h 

Jl X ' r £ _!=l JE ' r t =\ 

Continuing the above argument we have 
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U(n;j uj2 , ■■■ ,j e ) -dL{— [I a ri , — [[ a jl+r2} , — [[ ^-\ jt+ri 

J1 ' ri=l J r 2 =l J r^=l 
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That is, for 1 < i < d, j 1 ,j 2 , ■ • • ,i<eN with ji + j 2 H h # = d, 



(6.13) 

U{n;j u h, ■ ■ ■ -^(^n^^n a n+r 2 , II a ZtzUt+n 

JL ' n = l J2 ' r 2 = l J r e =l 

Thus using (16.1 3D we have 

d 

(6.14) P(n;a 1 ,a 2 , ■■■ ,a d ) ^ ^ (-1)' -I l7(7i; ii,j 2 , • • • 

3*1 H l-J>= d 

Next using Lemma 14.9( 1). for any ji,j2, • ■ ■ JjGN with j\ + h, • • • + je < d 
we have U(n]j u ■ ■ ■ ,j e ) /f~5 =1 J ' W is equal to 

c/(«;j,,---,^)r<|r;*(«)l 



d (^(^iir- - Jt) - \U(n;jx,--- ,ie)"|)uj 
U{nJ u --- ,j t ) - \U(n;ji,--- JifUx 

«§::>)- E 



if +1=1 



which is equal to 

U(n;ji,j 2 ,--- Jt) ~ \U(n;jx,j 2 ,--- J e )\jx 

t=l t=l j£+l d _ V j t 

li j n«f. 4 - e o;:,) n«f. t / - <»> 

t=i ri+i=l t=1 j£+i = l n +1 =l t=1 t=i 

which is 

-d [U(n;j 1 ,j 2 , ■■■ ,jt)- \U{n;ji,j 2 , ■ ■ ■ Jt)})* 



e+i 

r ,_ sr t d ~ E it d - E it- 1 . e+i 

" Jt t=l (=1 , , j£+l j V" 1 i\i 

t=l 1 — _ ^ — v Tl- ,e+1 -r-r 2^ Jt 



n'> « - V - — r ] t <> . / 



v* • m - -i E it+ r £+i , _-, „ _, E it+»*£+i l+y j t 

(d—2_jjt)i r *+i =1 *=i m+i=i t= i t =i 



t+i 

4=1 d- E it 



U{n;hi--- ,ji,d-^2j t ) - u (n;ji,--- JeJe+i)f l+l (n). 

t=i M +1 =i 1+ E i* 
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Using the fact and Lemma [4.9( 1). we have 

d-l 

zf(n) ~ d 4(n) = xi(n) - £ x?(n)f^(n) 

d-l 

= Q)ana 2 ■ ■ ■ a d - Ql)" 1 " 2 ' ' ' a hft+h W 

d-l 

~ d U{n-d)-Y,U{n-h)f^{n) 

d-l d-n-l 

^ d U{n-d)-[^ U ^^d- 3l )- £ U{n-j u j 2 )fl-^\n))). 
ii=i 32=1 
Continuing this argument we obtain 

d 

£=l h>- 'Hen 

Combining this with ( I6.14p . we have proved the Claim. 

7. Proof of Theorem C 

In this section we will prove Theorem C. That is, we will show that for d G N and 
F G ^GP d i there exist a minimal (i-step nilsystem (X, T) and a nonempty open set 
U such that 

FD{neZ:Un T~ n U Pi ... Pi T~ dn U + 0}. 
Let us explain the idea of the proof of Theorem C. Put Md = {B C Z : there 
are a minimal <i-step nilsystem (X, T) and an open non-empty set U of X with 
B D {n G Z : f|f =0 T _in C/ ^ 0}}. Similar to the proof of Theorem B(2) we first 
show that J^GPa ^ M d if an d only if {n G Z : | |P(n; «i, • • • , a^) 1 1 < e} G M d for any 
ai, • • • , a d G R and e > 0. We choose (X, T) as the closure of the orbit of Y in G d /T 
(the nilrotation is induced by a matrix A G G d ), define U <Z X depending on a given 
e > 0, put S = {n G Z : Hi=o T~ m U ^ 0}; and consider the most right-corner entry 
zf{m) in A nm BC m with B £ G d and C m G T for a given n £ S with 1 < m < d. 
We finish the proof by showing S C {n G Z : | |P(n; «i, • • • , ad) 1 1 < e } which implies 
that {n G Z : ||P(n;ai, ••• ,a d )|| < e} G jV d . 

7.1. The ordinary polynomial case. To illustrate the idea of the proof of The- 
orem C, we first consider the situation when the generalized polynomials are the 
ordinary ones. That is, we want to explain if p(n) is a polynomial of degree d with 
p(0) = and e > 0, how we can find a <i-step nilsystem (X, T), and a nonempty 
open set U C X such that 

(7.1) {n G Z : p(n) (mod Z) G (-e, e)} D {n G Z : [/ n T" n C/ PI ... Pi T^f/ ^ 0}. 
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To do this define T a>d : T d — > T d by 

T a ,d(9i, 9 2 ,..., 9 d ) = (9 1 + a, 9 2 + 0i, 9 3 + 9 2 , . . . , 9 d + 9a-i), 
where a G R. A simple computation yields that 

1 d 

(7.2) T^ d (9 u ...,9 d ) = (9 1 + na, n9 x + 9 2 + -n(n - . . . , ^ (^Jfc), 

i=0 

where O = a, n G Z and (£) = 1, (™) := u ^- j) for i = 1, 2, ■ • • , d. 

We now prove (17. ip by induction. The case when d — 1 is easy, and we assume 
that for each polynomial of degree < d — 1 (17. ip holds. Now let p(n) = Yut=i a i n% 
with OLi G K. By induction for each 1 < i < d — 1 there is an i-step nilsystem (Xj, Tj) 
and an open non-empty subset C/j of Xi such that 

{ii6Z: atrj (mod Z) G (-|, |)} D {n G Z : ^ n 2r n £/< n . . . n T" dn ^ ^ 0}. 



By the Vandermonde's formula, we know 

/ 1 2 3 

1 2 2 3 2 



<2 \ 

d 2 



1 2 d ~ 1 3 d " 1 ... d^ 1 
^ d 3 d ... d d J 



\ 1 T 

is a non-singular matrix. Hence there are integers Ai, A 2 , . . . , A<j and A G N such 
that the following equation holds: 

/ \ 





/ 1 


2 


3 . 


. d 


\ 


( 


Ai 


1 


2 2 


3 2 . 


. d 2 






A 2 


1 


2 d-i 


3 dLl . 


. d d ~ l 






Ad-i 


V i 


2 d 


3 d . 


. d d 


J 


\ 







V A/ 



That is, 



(7.3) 



^mm 3 = Ai + A 2 2 J ' + . . . + X d d j = 0, 1 < j < d - 1; 

m=l 
d 

^ A m m d = Ax + A 2 2 d + . . . + X d d d = A. 



m=l 



Now let T d = T^d d and = T d . Let K d = d\ Y? i= i M, d > with fQei < e/d and 
f/d = (-ei,ei) d . A ' 

It is easy to see that if n G {n G Z : C/ d n T d n U d T~ dn U d ^ 0} then we know 

that there is (6>i, . . . , d ) G £/ d such that T l d n (9i, . . . ,9 d ) G £7^ for each 1 < i < d. 
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Thus, by ( I7.2p considering the last coordinate we ge that 

Oo + (AK + ■ ■ • + (o)6d (mod Z) G (-ci, ei) 
O d o + (j^i + ■■■ + O0 d (mod Z) e {-e x ,e x ) 

O°o + {tx)°i + ■■■+ Co> ( mod Z ) e (-ei, ei), 

where # = Multiplying (™)#o + (^J^i + • • • + Oo)^ by Aj<i! and summing 
over i = 1, . . . , d we get that 

d d 

J2 X i dl J2(d^) e i = a dn d (mod Z) G (-K^K^) C (-e/d,e/d). 

j=l i=0 

Choose Xj G £/j for 1 < z < cf. Let x = (xi, X2, • • • , x^) G Xi x . . . x Xd and X be 
the orbit x under T = 7\ x T 2 . . . x T^. Then (X, T) is a d-step nilsystem. If we let 
U = (Ui x U 2 x . . . x U d ) n X, then we have (171]) . 

By the property of nilsystems and the discussion above it is easy to see 

Remark 7.1. Let k EN, qi(x) be a polynomial of degree d with ^(0) = and ej > 
for 1 < z < k. Then there are a d-step nilsystem (X, T, //) and B C X with /i(-B) > 
such that 

p|{n G Z : ||fc(n)|| < e*} D {n E Z : fi(B fl T~ n B Pi ... Pi T~ dn B) > 0} 
7.2. Some preparation. For d E N, define 

Md = {B C Z : there are a minimal ci-step nilsystem (X, T) and an open 

non-empty set U of X with B D {n E Z : U f) T- n U fl . . . fl T- d "[/ ^ 0}.} 
Hence Theorem C is equivalent to 

Lemma 7.2. For eac/i dGN, M d is a filter. 

Proof. Let B\,B 2 E Md- To show Md is a filter, it suffices to show B\ fl B 2 E Md- By 
definition, there exist minimal d-step nilsystems (Xj,Tj),z = 1,2, and a nonempty 
open set £/j such that 

BiD{neZ:U t n Tr n Ui n . . . n ir**^ ^ 0}. 

Taking any minimal point x = (xi,X2) G X\ x X2, let X = orb(x,T), where 
T = T\ x T 2 . Note that (X, T) is also a minimal d-step nilsystem. 

Since (Xj, Tj),i = 1,2, are minimal, there are fcj G N such that Xj G TV~ 
i = 1, 2. Let £7 = (Tf fel f/i x T^ k2 U 2 ) n X, then £7 is an open set of X. Note that 

{nez-.un T- n u n . . . n r- rf "[/ ^ 0} 

= f| {n G Z : T^EZ; fl ^ (tl+ ">[/, n . . . n T -^+ dn ) Vi ^ 0} 

i=l,2 

= p| {n e z : Ui n n . . . n ir d "^ ^ 0} 

i=l,2 
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Hence 

B x n B 2 d {n e z : u n T- n f/ n . . . n T- dn [/ ^ 0}. 

That is, 5i (1 B 2 G A/d and A/d is a filter. □ 
Definition 7.3. For r G N, define 

GP r = {p(n) G GP r : {n G Z : p(n) (mod Z) G (-e, e)} G AC, Ve > 0}. 

Remark 7.4. It is clear that for p(n) G GP r , p(n) G CP,, if and only if —p(n) G GP r . 
Since Af r is a filter, if Pi(n) , p2{n) , ■ ■ ■ ,Pk(n) G GP r then 

Pi(ra) + P2W H Vp k {n) e GP r . 

Moreover by the definition of GP r , we know that Tgp t C A4 if and only if GP r = 
GP r . 

Since we will use induction to show Theorem C, thus we need to obtain some 
results under the following assumption, that is for some d > 2, 

(7-4) Fap^QMd-i. 

Lemma 7.5. Let p(n) , q(n) G GPd with p(n) ~d (?(n). Under the assumption (17. 4p . 
p(n) G GPd z/ and on/y if q(n) G GPd- 

Proof. It follows from Lemma [6 .3[ A/d being a filter and J r GP d ^ 1 Q A/d-i C A/d- D 

Theorem 7.6. Under the assumption ( 17. 41) . the following properties are equivalent: 

(1) ^ G p d CAC. _ 

(2) P(n; ai, a 2 , • • ■ , «d) G GPd /or any a±, a 2 , ■ ■ ■ , ad G R ; £/iat is 

{n G Z : P(n; a ls a 2 , • ■ ■ , a d ) (mod Z) G (— e, e)} G Ad 

for any a\, a^, ■ ■ ■ , «d G K anc/ e > 0. 

(3) SGPd C GP d . 

Proof. The proof is similar to that of Theorem 16.71 □ 

7.3. Proofs of Theorem C. Now we prove J~GP d Q A/d by induction on d. When 
d = 1, since Tgp x — Fsgpx and A/d is a filer, it is sufficient to show that: for any 
p{n) = an G SGPi and e > 0, we have 

(7.5) {n G Z : p(n) (mod Z) G (-e, e)} G A/i- 

This is easy to be verified. 

Now we assume that for d > 2, J r cp d - l Q Ad-i, i.e. (17. 4p holds. Then it fol- 
lows from Theorem 17.61 that under the assumption (I7.4p . to show J~GP d Q Ad, it is 
sufficient to show that 

P{n;/3 1 ,f3 2 ,...,f3 d )eGP d , 

for any Pufa,-- , Pd G E. 

Fix /?!, /3 2 , • • • , /3d G K. We divide the remainder of the proof into two steps. 
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Step 1. We are going to show 



P(n; fa, / V", fa) -d E E (-l) /_1 ^(n; Ji, j 2 , • • • , jt), 



=1 3i,-j<eH 

31 H hj'^=d 



where as in the proof of Theorem B, we define 



(7.6) %ii) = -JJ«H l<3i<d. 

3 1 ' r=l 

And inductively for £ = 2, 3, • ■ ■ , d define 

U(n;j 1 ,j 2 ,--- ,je) = (U(n;j ir -- - [t/"(n; ji, • • • n a Et=ii*+r 



(^7(n;ji,-- - - |77(7i;ji,--- ,^_ 1 )])L(— jj «j^J 3 - 



J * r £ =l 

for ji, j2, • • • ,je > 1 and ji + • • • + ji < d (see (14.11) for the definition of L). 
In fact, let Ai, A 2 , • • • , A<j 6 Z and A G N satisfying 07.31) . Put 
"i = #l/A, a 2 = 2 , a 3 = /3 3 , . . . , a d = d - 

Then 

P(n; 0i, 02,---, fid) = AP(n; a u a 2 , • • • , 
Note that in proof of Theorem B we have 

d 

(7.7) P{n;a x ,a 2 , ■■■ ,a d ) E ( -1 )'~ lc/ '( n ;^i>i2> " " ' 



£=1 

31 H h3f=d 



Since A is an integer, we have 

d 

AP(n; an, a 2 , ■ ■ ■ , a d ) ~ d ^ E (- 1 Y~ 1 * U ( n 'i3^3^ • " " ,3t)- 

3i H hjf =d 

That is, 

P(n; ,0 d ) ~ d J2 E (-l)' _1 A«7(7i;ii,j 2 ,... 



= 1 3l.---3 f SN 
31 H h3£=d 



Hence, by Lemma [731 to show P(n; 0\, 2 , • • • , d ) G GPd, it suffices to show 

d 

( 7 - 8 ) E E (-i/^jbjv-j^GPrf. 



£=1 31,-'-3>eN 
31 H l-3> = f 
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Now choose x = )i<jfe<d,i<i<d_fc+i e R d( - d+1 ^ 2 with x\ — a.i for i — 1, 2, • • • , d 
and xf = for 2 < k < d and 1 <i < d — k + 1. Let 



A = M(x) 



/ 1 ai 

1 a 2 





\ 



\ 




a<i-i 
1 a<i 
1 / 



For ra G N, if x(n) = (x^ (n))i< fc < dj i<;< d _ fc+ i G M'W)/ 2 satisfies M(x(n)) = A", 
then by Lemma [6. II and Remark I6.2[ 

(7.9) x k (ri) = (l)aia i+1 ■ ■ ■ a i+k _i 

for 1 < k < d and 1 < i < d - k + 1. 

Let X = Gd/r be endowed with the metric p in Lemma 13.71 and T be the nil- 
rotation induced by A G G^, i.e. I?r i-)- ABY for £> G G^. Since G^ is a <i-step 
nilpotent Lie group and T is a uniform subgroup of G^, (X, T) is a d-step nilsystem. 
Let xq = T G X and Z be the closure of the orbit orb(xo, T) of e in X. Then (Z, T) 
is a minimal <i-step nilsystem. We consider p as a metric on Z. 

Step 2. For any e > 0, we are going to show there is a nonempty open set U of Z 
such that 

d 

{neZ:Y, E (-l/" 1 Af/(n;j a ,j aj ---,^)(modZ)e(-6,e)} 

(7.10) £=l ji.-^es 



31 H h7£=d 



3{ii6Z:(/n T~ n [/ n . . . n T~ dn u ^ 0}. 



That means £ E (-l) /_1 A?7(n; ji, j 2 , • • • ,jt) e GP d . 



^ d d 

Fix an e > 0. Take e 2 = min < -3— : , - k where K = > |A m | ( > m ; 

l2K(Eto^) 4/ d ~i ^ 

and let ei > be small enough such that e ei+e i + - +e i — 1 < e 2 . Let 

U = {z G Z : p(z, x ) <ei} = {cT eZ : p(cT, T) < a}. 

and let 

S = {n G Z : U n T _n C/ D ... PI T~ dn U + 0}. 

Now we show that 
<2 

5 C jn G Z : ^ ^ (-l^ALtyi; j 1; j 2 , • • • ,j t ) (mod Z) G (-e, e)} 

21 H hj£ =<i 

Let n & S. Then [/ n T~ n £/ D . . . D T~ dn £7 ^ 0. Hence there is some B e G d with 

Bret/n T" rt f/ n . . . n T~ dn u. 
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Thus 

p(A mn BT, r) < e u m = 0, 1, 2, . . . , d - 1. 

Since p is right translation invariant, we may assume that p(B,I) < e\, where I is 
the (d + 1) x {d + 1) identity matrix. 

For each m G {1, 2, ... , d}, since p(A mn .E>r, T) < e\ there is some C m G T such 
that 

(7.11) p{A mn BC m J)<e x . 

Let A m "5C m = M(z(m)), where z(m) = (^M^^i^xd-fe+i G R d ( d+1 )/ 2 . Then 
From ( 17. lip , we have \ \A mn BC m — I\ \ < e 2 by Lemma IXBl thus 

\z*(m)\<e 2 , l<k<d,l<i<d-k + l. 

On the one hand, since |zf(m)| < e 2 , we have 

d 

(7.12) ^A m z?(m) G (-Ke 2 ,Ke 2 ). 

m=l 

On the other hand, we have 



a a 

£ Ktfim) ^(^(-l)'- 1 W(ni 3i, h, ■ ■ -,3i)) 

(7 13) m=1 1=1 hJ2,-,jieN 

K ' ' h+32+—+3i=d 

+ A ( y (d + d 2 + ... + d d - 1 )(2Ke 2 )y 

Note that for a, b G R and 5 > 0, a « b+ A (5) means that a — b (mod Z) G (—5, 5). 

Since the proof of (17.131) is long, we put it after Theorem C. Now we continue the 
proof. By (1713]) and (I732]) . we have 

d 

^(-l)'- 1 £ XU(n;jx,j 2 ,...Jl) (modZ) e ( - M(2Ke 2 ), M{2Ke 2 )) C (-e,e), 

1=1 JiJa,— 

where M = 1 + d + . . . + d d ~ x . This means that 

d 

ne[qeZ:J2 (-^ l '^U(q; jx, j 2 , . . (mod Z) G (-e, e)}. 

3l+32+—+3l=d 

Hence 

d 

5 C [q G Z : ]T ^ (-l^A^g; j x , j 2 , . . . ,ji) (mod Z) G (-e, e)}. 

ji+32+-+ji=d 



d 

\£-l 



Thus we have proved (I7.10P which means ^2 12 ^U(n;jx,j 2 , ■ ■ ■ ,je) G 

1=1 h 
h- 

GPd- The proof of Theorem C is now finished. 



=1 ii.-ifGN 
JlH Hj>= d 
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7.4. Proof of (P7TT3|) . Since p(B, 7) < e 1; by Lemma EH 

(7.14) US- J|| < e ei+e ? + - +e ? - 1< e 2 < 1/2. 

Denote £ = M(y), where y = (y$)i< k <d,i<i<d-k+i G M^ +1 )/ 2 . From (JZH, 

< £2, 1 < fc < d, 1 < i < d - k + 1. 
For m = 1, 2, ■ • • , m, Recall that C m G T satisfies 

(7.15) p{A mn BC m ,I)<e x . 

Denote C m = M(h(m)), where h(m) = (-/i?(™))i<k<«i,i<»<d-*+i G Z<W)/ 2 . From 
(I7.15p . we have 

||A™flC m -I||<e 2 , m =l j2 ,...,d. 

Let A mn £? = M(w(m)), where w(m) = K fc (m)) 1 < fc < djl < i < d _ fc+1 G ]R d ( d+1 )/ 2 . Then 

fc-i 

w *(m) = xf(mn) + (^a^(mn)yj^) + y • 



<i 
3=1 

k-l 

, k ~3 _L o. k 



In i r>\ (mn\ , \ v fmn\ k—j . jfe 

(7.16) = ( fc Ja^-H • ■ ■ a i+ fc-i + { j ) a i a i+i " " " a i+j vVi+j + V% 



a \jnnj 

~ Gin ... Gt 



+k-l + 



fc! 

3=1 

where m = 1, 2, . . . , d, af(j') does not depend on m and |af (0)| = \yf\ < e 2 . 
Recall that z(m) = (2^(777.) Ji^^i^j^-jt+i G 

R d(d+i)/2 satisfies 

M(z(m)). Hence 

fc-i 

(7.17) **(m) = w*{m) - (^^M^/M) - fc*(m)- 

i=i 

From ||A mn SC m - J|| < e 2 , we have 

|zf(m)|<e 2 , 1 < £; < d, l<i<d-k + l. 
Note that hf(m) G Z, and we have 



fc-i 

fc-j 



fr* ( TO ) = (m) - (to) h\ 

3=1 



Let 



fc-i 



3=1 



Then 

\u k (m)-h^m)\ = \z^m)\<e 2 < 1/2. 
Recall that for a, b G R and 5 > 0, a ~ b+ A (5) means a — b (mod Z) G (—5, S). 
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Claim: Let 1 < r < d — 1 and v r (0), v r (l), . . . , v r (r) G R. TTien for each 1 < r x < 
d — r — 1 and 1 < j < r\ + r, there exist v r>ri (j) G K such that 

(1) 

d r 

^A™^™ 1 ^^))^;^™) « A(v r (r) - \v r (r)~}) ™_ a 1+r . . . a d 



(d-r)\ 

E E^X>Vi(0)*&£("0 + A (2iTe 2 ) 
ri = l m=l t=l 

(2) f rjri (r + r\) = [v r — \v r ]J — j-av+i . . . a r+ri /or 1 < r\ < d — r — 1. 



m=l t=0 

d— i — 1 d ri+r 



Proof of Claim. Since \uf^.(m) — hf^.(m)\ < e 2 , we have 



^A^mU)- MOD) | < X>™l (][>') 

m=l t=0 

<(Ew)(E m 1^' 



m=l t=0 m=l t=0 

d 



m=l t=0 



Hence 



CM 



m=l t=0 
d r 

(7.i8) « E ^(ZXMt) - Mf)l))/CM 

m=l t=0 
d r 

]T A m (^m*K(t) - K(t)]))<;(m)+ A (tfe 2 ). 



m=l t=0 

Then we have 

d r 

£ A m (J» r (o - m*)1))<;J» 

m=l t=0 

d r d— r— 1 



^A m (^m> r (o-r^wi))(<;M- E ^r+rHfta^H). 

m=l t=0 ri=l 
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From (I7.16P we have 

.d- 



Z A ™(Z m 'M*) - 



+r\ m ) 



m=l t=0 

d r / \d— r c(— r— 1 

= Z) A " ( ZX M*) ~ K (t)] ) ) ( j r > ^i+r ■ ■ ■ a d + ^ rniai+lti)) 

m=l t=0 ^ >' j=0 



-.d—r 



Z ^ mmd (d- )\ ai + r ■ ■ ■ a d(yr{t) ~ \Vr(t)\) 
m=l ^ ' 

d-1 d / 



h=l m=l \ 0<t<r 

0<j<d-r-l 
t+j=h 



+Z A -H°)- r*r(o)i)o^(o), 

m=l 

and so ^^^(ELo™'^^ - WODjuftfa) is equal to 

d—r d 

AT^^ai+r-.-adW*)- k-(*)l) + (Z A ™)M°)- K(o)i)^; 

^ ' m=l 



A (t/ _ r )i ai+r ' ' -^M*) _ M*)~l)+ A (Ke 2 ). 



The last equation follows from 



d 



|(Z A ™)Mo) - r^(o)l)^; r r | < Z l A -l £2 < ^ e2 - 

m=l m=l 

Then for 1 < r± < d — r — 1, by (17.161) . we have 

d r 
m=l t=0 

d r , s r ri— 1 

= Z Am (Z m< M0 ~ M*)"l))( 7 \\ a 1+r ...a r+ri + 2^m J a^ r (j)J^ 

m=l t=0 \ 1 >' 

d _ 

Z A ™( mr+ri "7jryj" a i+r • • -ar+n(^r(i) - K-(01)+ 

r+J*i — 1 

Z ^ (^(*)-K(t)l)a? + ,(j)))fc^H- 



m=l 

r+ri — 1 



h=0 0<t<r 

0<j<rx-l 
t+j=h 
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Let 

v r , ri (h) = E M*) - MW^i+rti), 0<h<r + n -l; 

0<t<r 
0<j<r±—l 

t+i=h 



[n 



v rri (r + ri) = — — T a 1+r . . . a r+ri (v r (t) - \v r (t)]) ; 

(n)! 

v r , ri (0) = (v r (0) - \v r (0)]) a[\ r (0) = (v r (0) - \v r (0)])y[\ r (0)- 
It is easy to see that |tv iri (0)| < e-i- Then 

d r 

m=l i=0 

d r+ri 

= E A -( E AnWjC:^)- 

m=l t=Q 

To sum up, we have 

d r 

E A «> ( E m * («•■ (*) - r«r ) ) m 

77) =1 i=0 

d— r— 1 / d ri+r 



A 



(n) d - r 



3yai +r . ..Q! d (t> r (t) - \v r (t)]) - E f E Am ( E m *^^i(*)) /l i4+t-? ) ( TO ) 



(d-r)l 



+ A (Jfe 2 ). 

Together with ( 17.18ft , we have 

E A m(E m * Wr (*0^+'' <TO ' ) ~ \d- -y Ql+ r .-aj(»r(f) ~ |«r(*)l 



m=l 4=0 

d—r—1 / d ri+r 



(d-r)! 

E ( E A -(E w V 1 (*))^? 3 (m)VA(2i^e 2 ). 



n=l \m=l 4=0 / 

The proof of the claim is completed. □ 
We will use the claim repeatedly. First using ( I7.17P we have 

d d , d-1 



a a / a—L 

E KAipi) « ^ A m U?(m) - ]T <(m)/^(m) 

m=l m=l ^ il=l 

By (I7.16p . we have 



d d ^ d 

^ \ m wf(m) « A m m d — ai . . . a d + A m yf 

m=l m=l m=l 



A— «! . . A (ife 2 ). 

a! 



7-' 1 
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Using this, (I7.16P and the claim, we have 

d d d d— 1 

^ A m zf(m) « A^-ai ...a d - ^ A TO ^ (m J1 ^-j-ai ...a^ + ^ m'aj 1 (*))/i?+£(m)+ A (Ke 2 ) 

m=l ' m=l 21=1 ^ t=0 

~ A H ai ■ • ■ ° d ~ 1 E A (d- ji)! ai+J ' 1 ■■■ arf (— |- ai '-- ajl ~ r-rai---ajilj 
+ 2^ L A « ™ J J (-y"! ■ • ■ Q 3i " f-7 a i ■ ■ • "jil J -j-ai+ii ■ • ■ Q ji+j2 

21=1 22=1 \m=l V n ' n ' j2 ' 

+ ^E rn\ lda (t))h^^'\m))+ A ((2(d-l)if + if)e 3 ). 



Note that here we use t> ^(t) = erf (t), t = 0, 1, . . . , j± — 1 and Vj x {j\) = ^t«i • • • ol 
Recall the definition of U(-): 

—ai ...a d = U(n; d), 

—a x . . . a h - \—ai . . . a h ] ) —ra l+h . . . a jl+h = U{n;j u j 2 )- 
Substituting these in the above equation, we have 

d d-1 

A m zf(m) « \U{n:d) - ^ XU ( n: . 3u d - ji) 

m=l jl=l 

d-l d-ji-l / d / h+h-1 \ ^ 

+ E E EM™ Jl+J2C/ (^i^)+ E rn% ld2 (t))hi-^+i 2 \m) 

21=1 22 = 1 \m=l \ t=0 ' / 

+ A {2dKe 2 ) 
Using the claim again, we have: 

d d-l d-l d-3i-l 

2J A ro ^i(m) « AJ7(n;d) - 2J AJ7(n;ii,i 2 ) + E E xu ( n \h-,j2,d-ji-h) 

m=l 2i=l 2i=l 22 = 1 

d-l d-ii-1 d-(2i+22)-l / d / 

-EE E [Y, x A mh+j2+j3U ^^32,h)+ 

21=1 22 = 1 23=1 \m=l 

21 +.72+23-1 x \ 

E '" ; ''.r.,,.M (*) ^SSf } H + A ( 2 ^2 + 2rf 2 ^e 2 ). 
t=o ' / 

Inductively, we have J^ =1 X m zf(m) 



Et- 1 )'" 1 E AC/(n;j 1 ,j 2 ,...,j / ))+ A (2dife 2 + 2d 2 Xe 2 + ... + 2^-^62) 

*J=i j'i,j2,-,iieN 

J1+J2+— +ii=d 

d 

E mn;h,h,...,ji))+ A ((d + d 2 + ... + rf d - 1 )(2^e 2 )^ 
J=i ji,i2,...,j'ieN 

21+j2 + ...+2!=d 

The proof of (17.131) is now finished. 
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8. Applications 

Our main results can be applied to get results in the theory of dynamical systems. 
As the limitation of the length of the paper, here we only state the results and the 
detailed proofs will appear in a forthcoming paper by the same authors. 

8.1. d-step almost automorpy. The notion of almost automorphy was first intro- 
duced by Bochner in 1955 in a work of differential geometry [7J [8]. Veech showed 
that each almost automorphic minimal system is an almost one-to-one extension 
of a compact metric abelian group rotation [28]. Let (X, T) be a minimal system 
and d G N. (X, T) is called a d-step almost automorphic system if it is an almost 
one-to-one extension of a <i-step nilsystem. Let it : X — > Y be the almost one- 
to-one extension with Y being a <i-step nilsystem. A point x G X is d-step almost 
automorphic if 7r _1 7r(x) = {x}. 

Using the main results of this paper, we show that Tp Q i d -, ^Bir d and 7^,0 can be 
used to characterize ci-step almost automorphy, i.e. in some sense, Tp % d and J"2 
can not be distinguished "dynamically". Similar results can be found in the next 
subsections. 

Theorem 8.1. [22] Let {X,T) be a minimal system, x G X and d G N. Then the 
following statements are equivalent 

(1) x is d-step almost automorphic. 

(2) N(x, V) G Fpoij for each neighborhood V of x. 

(3) N(x, V) G J^Boia f or eac h neighborhood V of x. 

(4) N(x, V) G Fd,Q for each neighborhood V of x. 

8.2. Regionally proximal relation of order d. 

8.2.1. Regionally proximal relation. Regionally proximal relation plays a very im- 
port role in the theory of topological dynamics. It is the main tool to characterize the 
equicontinuous structure relation S eq (X) of a system (X, T); i.e. to find the small- 
est closed invariant equivalence relation R(X) on (X, T) such that [Xj R(X),T) is 
equicontinuous. Veech [29] gave the first proof of the fact that the regionally proxi- 
mal relation is an equivalence one. Also he showed that Poincare sets can be used 
to characterize regionally proximal relation. 

8.2.2. Regionally proximal relation of order d. In [17J Host and Kra defined a d-step 
nilfactor for each ergodic system, see also [31J. To get a similar factor in topological 
dynamics Host, Maass and Kra introduced the notion of regionally proximal relation 
of higher order. 

Definition 8.2. J20J [19] Let (X, T) be a system and let d > 1 be an integer. A 
pair (x, y) G X x X is said to be regionally proximal of order d if for any 5 > 0, 
there exist x', / el and a vector n = (m, . . . , rid) £ ^ d and e G {0, l} d such that 
d(x,x r ) < S,d(y,y') < 5, and 

p(T n 'V, T n V) < 5 for any e ^ (0, 0, . . . , 0), 

where n • e = n\t\ + . . . + n^d- 
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The set of regionally proximal pairs of order d is denoted by RP' (X), which is 
called the regionally proximal relation of order d. 

It is easy to see that RP' d '(X) is a closed and invariant relation for all d G N. 
When d — 1, RP' d '(X) is nothing but the classical regionally proximal relation. In 
[19] , for distal minimal systems the authors showed that RP' d '(X) is a closed invari- 
ant equivalence relation, and the quotient of X under this relation is its maximal 
cf-step nilfactor. Recently, these results are shown to be true for general minimal 
systems by Shao-Ye [27]. We remark that a point is <i-step almost automorphic if 
and only if RP' d '[i] = {x}. Moreover, we have the following theorem whose proofs 
are based on the results built in this paper. 

Theorem 8.3. (22] Let {X,T) be a minimal system and d G N. The following 
statements are equivalent: 

(1) (x,y)eBPW{X) 

(2) N(x, U) G Fpoia for each neighborhood U of y. 

(3) N(x, U) G J^Birj for each neighborhood U of y. 

(4) N(x, U) G J-^o for each neighborhood U of y. 

We remark that for d = 1 the above theorem is known, see for example [2HJ El EE] • 

8.3. NiLj Bohro sets and sets SGd{P)- In this article to study NiLj Bohro sets, 
we use generalized polynomials. In [18], Host and Kra introduced an interesting set 
(SGd set) to study the Nil^ Bohr sets. Here we state some results and give some 
questions. First we recall definitions introduced by Host and Kra in pjB] . 

8.3.1. Sets SGd(P) and SG d . Let d > be an integer and let P = {pi}i be a (finite 
or infinite) sequence in N. The set of sums with gaps of length less than d of P is 
the set SGd(P) of all integers of the form 

eiPi + e 2 £>2 + • • • + e n p n 

where n > 1 is an integer, ej G {0, 1} for 1 < i < n, the are not all equal to 0, 
and the blocks of consecutive 0's between two 1 have length less than d. A subset 
A C N is an SG d -set if A fl SGd(P) ^ for every infinite sequence P in N. 

Note that in this definition, P is a sequence and not a subset of N. For example, 
if P = {pi,P2, ■ ■ ■}, then SG\(P) is the set of all sums p m + p m +i + • • • + Pn of 
consecutive elements of P, and thus it coincides with the set A (S) where S = 
{PiiPi + P2,Pi + P2 + P3, ■ ■ ■}■ Therefore S'G^-sets are the same as A*-sets. 

In [TS] a notion called strongly piecewise-F, written PW- J 7 was introduced and 
the following proposition was proved 

Proposition 8.4. Every SG* d -set is a PW-Nil d Bohr -set. 

Host and Kra asked the following question. 

Question 8.5. Is every Nild Bohr -set an SG* d -set? 

Though we can not answer this question, we show that they can not be distin- 
guished dynamically (see Theorems 18.31 and 18.71) . Using Theorem B, Question 18.51 
can be reformulated in the following way: 
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Question 8.6. Let deN and S be an SG^-set. Is it true that for any k G N, any 

Pi, . . . , Pfc G -PsGP d an d an V e i > 0, there is n G 5 sttc/i £/ta£ 

Pi(n)(mod Z) G (-e^e*) 

/or alii = 1, . . . ,k? 

We remark that since a (i-step nilsystem is distal, the above question has an 
affirmative answer for any IP-set. 

8.3.2. SGd sets and regionally proximal relation of order d. We can use SGd sets to 
characterize regionally proximal relation of order d. For each d G N, denote by SGd 
the collection of all sets SGd(P), and J-sG d the family generated by SGd- 

Theorem 8.7. [22] Let (X,T) be a minimal system. Then for any d G N, (x,y) G 
RpH if an d on iy if N(x, U) G J~sG d for each neighborhood U of y. 

A direct corollary of Theorem 18.71 is: let (X, T) be a minimal system, i6l, and 
deN. If x is .FcJ^-recurrent, then it is d-step almost automorphic. Since J~sG d does 
not have the Ramsey property [22], we do not know if the converse of the above 
corollary holds. 
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